Method, system and apparatus for route optimization of large scale electric vehicle fleet operations

Abstract

The application discloses a path optimization method, system and device for large-scale electric vehicle fleet operation, and relates to the technical field of vehicle path planning.The application comprises the following steps: defining an electric vehicle routing problem with a time window based on a graph structure, describing the electric vehicle routing problem with a time window from the perspective of reinforcement learning; constructing an attention model for aggregating local and global information of the graph structure; using a random sampling method to sample the solution result of the graph structure; and finally training the attention model by using a rollout baseline policy gradient algorithm.The application can quickly and efficiently find the shortest path for the electric vehicle fleet under the condition of ensuring the position and time of the customer, make a routing decision, minimize the total distance of the vehicle fleet, and has great scalability and can support super large-scale electric vehicle fleet operation instances.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automobile path planning, in particular to a path optimization method, system and device for large-scale electric vehicle fleet operation. BACKGROUND

[0002] In the context of the electric vehicle routing problem with time windows, capable electric vehicle fleets are responsible for providing services for customers in a specific area, each customer has their own demand, and this demand needs to be met within a specified time, so more careful route planning and more accurate time prediction are needed to ensure that customer demand is completed within the specified time and the path with the minimum total distance is found.

[0003] Existing heuristic algorithms (VNS / TS) and precise branch-and-price-cut algorithms proposed according to four variants of the electric vehicle routing problem with time windows can provide high-quality solutions for benchmark instances of the electric vehicle routing problem with time windows, but the quality and efficiency of the solutions decrease as the size of the instance increases, and cannot solve large-scale electric vehicle routing problem instances with time windows. Therefore, we propose a path optimization method, system and device for large-scale electric vehicle fleet operation. SUMMARY

[0004] The purpose of the present application is to provide a path optimization method, system and device for large-scale electric vehicle fleet operation, which can quickly and efficiently find the shortest path for the electric vehicle fleet under the condition of ensuring the location and time of the customer, make routing decisions, and minimize the total distance traveled by the fleet. It can support super large-scale electric vehicle fleet operation instances and balance quality and efficiency.

[0005] According to the first aspect of the present application, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme: a path optimization method for large-scale electric vehicle fleet operation, comprising the following steps:

[0006] The electric vehicle routing problem with time windows is defined based on a graph structure, and the system state, reward scheme and masking strategy of the electric vehicle routing problem with time windows are described from the perspective of reinforcement learning;

[0007] An attention model is constructed, and the attention model is combined with a graph embedding component to aggregate local and global information of the graph structure;

[0008] A random sampling method is used to sample the solution results of the graph structure;

[0009] A rollout baseline policy gradient algorithm is used to train the attention model to obtain a trained attention model for finding the shortest path for the electric vehicle fleet.

[0010] Further, the time windowed electric vehicle routing problem is defined based on a graph structure, which is as follows:

[0011] (21) The weight of each edge of the graph is the Euclidean distance between the connected vertices. The graph structure includes three types of vertices: customers V c , stations V s and warehouses V d ;

[0012] At this time, the solution to the time windowed electric vehicle routing problem is the vertices in the graph structure, which is the planned route of the electric vehicle;

[0013] (22) Each vertex i is associated with an array , and all vertex arrays form an information set X t , which is used to decode the local information at the vertex at step t. In the above formula, x i and z i represent the geographic coordinates of vertex i, e i and l i represent the corresponding time window, represents the remaining demand of vertex i at decoding step t;

[0014] (23) The global variables shared by each vertex are:

[0015] G t ={τ t ,b t ,σ t}

[0016] In the formula, τ t represents the time of activity EV at the beginning of decoding step t, and the initial value is set to 0; b t represents the battery capacity, and the initial value is set to the battery capacity of the electric vehicle; σ t represents the number of available electric vehicles, and the initial value is set to the size of the vehicle fleet.

[0017] Further, the system state, reward scheme and masking strategy of the time windowed electric vehicle routing problem are described from the perspective of reinforcement learning, which is as follows:

[0018] (31) In the time windowed electric vehicle routing problem, the system state is represented by the information set X t and the global variable G t of the graph structure;

[0019] Suppose there is an agent that will add a vertex at the end of the current sequence according to the current system state and given information at each step, and the system state will change accordingly. Repeat this process until the termination condition is met and stop;

[0020] Assume that the process is terminated at step t m with termination condition that all customer demands are satisfied, define y t to represent the vertex selected at step t, define Y t to represent the sequence of vertices formed before step t;

[0021] At each decoding step t, given the set X t , the global variables G t , and the travel history Y t , estimate the probability of each vertex i being added to the sequence, and decode the next vertex y t+1 to visit according to this probability distribution, the probability distribution formula is as follows:

[0022] P(y t+1 |X t , G t , Y t )

[0023] where X t is the information set; G t is the global variable set; Y t represents the sequence of vertices formed before step t;

[0024] (32) Based on the obtained y t+1 , update the system state using the transition function formula (1)-(4), as follows:

[0025] (32.1) System time τ t+1 :

[0026]

[0027] where V c represents the customer, V s represents the station, and V d represents the warehouse; represents the maximum of the system time and the vertex; y t represents the vertex selected at step t, w(y t , y t+1 ) is the travel time from vertex y t to vertex y t+1 ; b t represents the battery power, re(b t ) is the time required to charge the battery from a given standard to full, and s is a constant representing the service time of each customer vertex;

[0028] (32.2) Battery power b t+1 :

[0029]

[0030] where b t represents the battery capacity, V c represents the customer, y t represents the vertex selected at step t, f(y t ,y t+1 ) is the energy consumption from the electric vehicle vertex y t to the vertex y t+1 , and B is the battery capacity;

[0031] (32.3) the number of electric vehicles σ t available at each vertex and the remaining demand ε t i Update:

[0032]

[0033] where V d represents the warehouse, y t represents the vertex selected at step t.

[0034] (32.4) Set the reward function, which is used to guide the attention model to generate a solution subject to conditions, the first term in the reward function is set as the negative total distance traveled by the vehicle fleet to support short distance solutions, and the other terms are penalty terms for problem constraints, as follows:

[0035]

[0036] where σ t represents the number of electric vehicles, b t represents the battery capacity, y t represents the vertex selected at step t, Y t represents the sequence of vertices formed before step t, w(y t-1 ,y t ) is the travel time along edge (y t-1 ,y t ), S is the number of site visits along the trajectory , and β1, β2, and β3 are three negative constants.

[0037] Further, an attention model is constructed, which is combined with a graph embedding component to aggregate local and global information of the graph structure, as follows:

[0038] The attention model includes a graph embedding component, an attention mechanism, and an LSTM decoder;

[0039] (41) the graph embedding component

[0040] First, the input information set X of the attention model t and the global variable G t Mapped to high-dimensional vector space, respectively recorded as and And use the Structure2Vec tool to synthesize the embedding vector;

[0041] For each vertex vector i, the vector Initialize and then update using the recursive formula After p rounds of recursion, the attention network model will generate a ξ-dimensional vector for each vertex i And set for

[0042] In each round of recursion, global information and position information are aggregated through the first two terms of the recursive formula, and information of different vertices and edges is propagated to each other through the last two summation terms, so that the final embedding vector It contains both local information and global information. The recursive formula is as follows:

[0043]

[0044] Where N(i) is the set of vertices connected to vertex i by an edge, and this set is called the neighborhood of vertex i. w(i,j) represents the travel time on edge (i,j). θ1, θ2, θ3, θ4, and θ5 are trainable variables. Relu is a nonlinear activation function, and relu(x) = max{0,x}.

[0045] (42) Attention Mechanism

[0046] Based on embedding vector Utilize the context-based attention mechanism to calculate the visit probability of each vertex i;

[0047] First calculate the context vector c t , the state of the entire graph structure is specified as the weighted sum of all embedding vectors, that is:

[0048]

[0049] The weight of each vertex is defined as follows:

[0050] a t =softmax(v t ) (8)

[0051]

[0052] Where a t context vector, For the i-th entry of the vector v t h t is the hidden memory state of the LSTM decoder, θ v and θ u are trainable variables, [;] denotes concatenation of two vectors on both sides of the symbol ";", tanh is a nonlinear activation function, is a normalized exponential function applied to a vector,

[0053]

[0054] Then, the probability of visiting each vertex i in is estimated as follows:

[0055] p t = softmax(p t ) (10)

[0056]

[0057] where p t is the context vector, is the i-th entry of the vector g t , θ c and θ g are trainable variables.

[0058] (43) Masking strategies

[0059] The following masking strategies are designed to exclude infeasible routes, as follows:

[0060] vertex j represents a customer whose unfulfilled demand is zero or exceeds the remaining goods of the electric vehicle;

[0061] vertex j represents a customer, and the current battery level b t of the electric vehicle cannot support the electric vehicle to complete the journey from vertex i to vertex j and then to the factory;

[0062] the earliest time to arrive at vertex j violates the time window constraint, i.e., τ t + w(i, j) > l j ;

[0063] the electric vehicle travels from vertex i to vertex j and cannot return to the warehouse before the specified time T ends;

[0064] the electric vehicle is currently in the warehouse and has no remaining goods at any customer vertex, then all vertices except the warehouse are masked;

[0065] (44) LSTM decoder: The decoder network is modeled using LSTM.

[0066] Further, the graph structure solution result is sampled by using random sampling method, and the specific process is as follows:

[0067] For all vertices i, at each decoding step t, the next vertex to be accessed is sampled according to the probability distribution described above, and the process is repeated to obtain multiple solutions of an instance, and the shortest distance solution is reported.

[0068] Further, the attention model is trained by using a rollout baseline policy gradient algorithm to obtain a trained attention model, which is used to find the shortest path for the electric vehicle fleet, and the specific process is as follows:

[0069] (61) Define the loss function

[0070] The loss function represents the negative expected reward of the sampled trajectory Y, and the formula is as follows: θ

[0071]

[0072] (62) Gradient estimation

[0073] Estimate the gradient of the loss function L(θ) with respect to the trainable variable θ, and the formula is as follows:

[0074]

[0075] In the formula, X t is the information set; G t is the global variable set; Y t represents the vertex sequence formed before step t; the parameter N is the batch size, X [i] is the i-th training example in the batch, Y [i] is the corresponding solution generated using π θ ; BL() represents the rollout baseline, P θ (Y [i] │X [i] ) represents the probability of generating the solution Y [i] using the random strategy π θ . [i]

[0076] According to the second aspect of the present application, the present application provides a path optimization system for large-scale electric vehicle fleet operation, which is used to realize the above-mentioned path optimization method for large-scale electric vehicle fleet operation, comprising:

[0077] A definition module is used to define the electric vehicle routing problem with time window based on the graph structure, and the system state, reward scheme and shielding strategy of the electric vehicle routing problem with time window are described from the perspective of reinforcement learning. ​​​

[0078] a construction module, configured to construct an attention model, and combine the attention model with the graph embedding component to aggregate local and global information of the graph structure;

[0079] a sampling module, configured to sample the solution result of the graph structure by using a random sampling method;

[0080] a training output module, configured to train the attention model by using a rollout baseline policy gradient algorithm to obtain a trained attention model, and find the shortest path for the electric vehicle fleet.

[0081] According to a third aspect of the present application, a terminal device is provided, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the path optimization method for large-scale electric vehicle fleet operation is adopted.

[0082] The present application has at least the following advantages:

[0083] 1. The present application can quickly and efficiently find the shortest path for the electric vehicle fleet under the condition of ensuring the position and time of the customer, make a routing decision, minimize the total distance of the fleet, and has great scalability, can support super large-scale electric vehicle fleet operation instances, and balance quality and efficiency.

[0084] 2. The attention model in the present application can quickly capture important information in the embedded graph and has higher solution efficiency in solving the electric vehicle routing problem with time window.

[0085] Of course, implementing any product of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0086] Figure 1 a flowchart of the optimization method described in the embodiment of the present application;

[0087] Figure 2 a structure diagram of the electric vehicle routing problem with time window in the embodiment of the present application;

[0088] Figure 3 a structure diagram of the LSTM decoder in the embodiment of the present application. DETAILED DESCRIPTION

[0089] With reference to the drawings of the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present disclosure.

[0090] Please refer to Figure 1 , Figure 2 and Figure 3 , the present application provides a technical solution: a path optimization method for large-scale electric vehicle fleet operation, comprising the following steps:

[0091] S1. Defining the electric vehicle routing problem with time window based on the graph structure, describing the system state, reward scheme and shielding strategy of the electric vehicle routing problem with time window from the perspective of reinforcement learning, as follows:

[0092] S11. Selecting to define the electric vehicle routing problem with time window on a graph structure, and the weight of each edge of the graph structure is the Euclidean distance between the connected vertices;

[0093] The graph structure includes three types of vertices: customers (V c ), stations (V s ) and warehouses (V d );

[0094] At this time, the solution of the electric vehicle routing problem with time window is the vertex in the graph structure, that is, the planned route of the electric vehicle;

[0095] Each vertex i is associated with an array , and all vertex arrays form an information set X t , which is used to decode the local information at the vertex at step t, wherein x i and z i represent the geographical coordinates of vertex i, e i and l i represent the corresponding time window, represents the remaining demand of vertex i at decoding step t; each vertex shares a set of global variables, and all global variables can change over time:

[0096] G t ={τ t ,b t ,σ t}

[0097] In the formula, τ t represents the time of the activity EV (electric vehicle) at the beginning of the decoding step t, and the initial value is set to 0; b trepresents the battery capacity of the electric vehicle, and the initial value is set as the battery capacity of the electric vehicle; σ t represents the number of available electric vehicles, and the initial value is set as the size of the vehicle fleet;

[0098] S12. The system state, reward scheme and shielding strategy of the electric vehicle routing problem with time window are described from the perspective of reinforcement learning, as follows:

[0099] In the electric vehicle routing problem with time window, the system state is represented by the information set X t and the global variable G t in the graph structure;

[0100] Assume that there is an agent that will add a vertex at the end of the current sequence according to the current system state and given information at each step, and the system state will change accordingly. Repeat this process until the termination condition is met and stop;

[0101] Assume that the termination occurs at step t m , and the termination condition is set to meet all customer demands. Define y t to represent the selected vertex at step t, and define Y t to represent the sequence of vertices formed before step t.

[0102] At each decoding step t, given the set X t , the global variable G t and the travel history Y t , estimate the probability of each vertex i being added to the sequence according to the probability distribution formula, and decode the next vertex y t+1 to be visited according to this probability distribution. The probability distribution formula is as follows:

[0103] P(y t+1 =i|X t ,G t ,Y t )

[0104] In the formula, X t is the information set; G t is the global variable set; and Y t represents the sequence of vertices formed before step t.

[0105] Based on the obtained y t+1 , update the system state using the transition function formula (1)-(4), as follows:

[0106] System time (τ t+1 ):

[0107]

[0108] In the formula, Vc V represents a customer, V s V represents a station, V d V represents a warehouse; y represents the system time and the maximum value of the vertex; y t y represents the vertex selected at step t, w(y t , y t+1 ) is the travel time from vertex y t to vertex y t+1 ; b t represents the battery power, re(b t ) is the time required to charge the battery from a given standard, and s is a constant representing the service time of each customer vertex;

[0109] Battery power (b t+1 ):

[0110]

[0111] In the formula, b t represents the battery power, V c represents a customer, y t represents the vertex selected at step t, f(y t , y t+1 ) is the energy consumption from the electric vehicle vertex y t to vertex y t+1 , and B is the battery capacity;

[0112] The number of electric vehicles available at each vertex σ t and the remaining demand Update:

[0113]

[0114] In the formula, In the formula, V d represents a warehouse, y t represents the vertex selected at step t;

[0115] A reward function is set to guide the attention model to generate a condition-constrained solution, the first term in the reward function is set as the negative total distance traveled by the vehicle fleet to support short-distance solutions, and the other terms are penalty terms for problem constraints, as follows:

[0116]

[0117] In the formula, σ t represents the number of electric vehicles, b t represents the battery power, y t represents the vertex selected at step t, Y t represents the sequence of vertices formed before step t, w(yt-1 y t ) is the travel time along the edge (y t-1 y t ), S is the number of visits to the site along the trajectory , β1, β2 and β3 are three negative constants.

[0118] S2. Construct an attention model, combine the attention model with a graph embedding component, and use the attention model to aggregate local and global information of the graph structure, as follows:

[0119] The attention model includes a graph embedding component, an attention mechanism and an LSTM decoder;

[0120] S21. The graph embedding component

[0121] First, map the input information set X t and the global variable G t to a high-dimensional vector space, respectively denoted as and and use the Structure2Vec tool to synthesize the embedding vector;

[0122] Initialize the vector for each vertex vector i, and then update After p rounds of recursion, the attention network model will generate a vector for each vertex i with dimension ξ, and set to

[0123] In each round of recursion, the global information and the location information are aggregated through the first two terms of the recursive formula, and the information of different vertices and edges is propagated through the last two summation terms, so that the final embedding vector contains both local information and global information, and the recursive formula is as follows:

[0124]

[0125] In the formula, N(i) is the set of vertices connected to vertex i through an edge, which is called the neighborhood of vertex i, w(i,j) represents the travel time on edge (i,j), θ1, θ2, θ3, θ4, θ5 are trainable variables, and relu is a nonlinear activation function, relu(x) = max{0,x};

[0126] S22. Attention mechanism

[0127] Based on the embedding vector , use a context-based attention mechanism to calculate the visit probability of each vertex i;

[0128] First, the context vector c is calculated t The state of the entire graph structure is specified as the weighted sum of all embedding vectors, i.e.,

[0129]

[0130] The weight of each vertex is defined as follows:

[0131] a t = softmax(v t ) (8)

[0132]

[0133] where a t is the context vector, is the i-th entry of the vector v t , h t is the hidden memory state of the LSTM decoder, θ v and θ u are trainable variables, [;] denotes concatenation of two vectors on both sides of the symbol ";", tanh is a nonlinear activation function, is a normalized exponential function applied to a vector,

[0134] Then, the probability of visiting each vertex i in is estimated, which is formulated as follows:

[0135] p t = softmax(p t ) (10)

[0136]

[0137] where p t is the context vector, is the i-th entry of the vector g t , θ c and θ g are trainable variables.

[0138] S23. Masking strategy

[0139] To speed up the training process and ensure the feasibility of the solution, the following masking strategy is designed to exclude infeasible routes, which is as follows:

[0140] Vertex j represents a customer whose unmet demand is zero or exceeds the remaining goods of the electric vehicle;

[0141] Vertex j represents a customer, and the current battery level b tIt is impossible for the electric vehicle to support the trip from vertex i to vertex j and then to the depot;

[0142] The earliest time of reaching vertex j violates the time window constraint, i.e. t +w(i,j)>l j ;

[0143] The electric vehicle travels from vertex i to vertex j (if vertex j is a station, it charges at vertex j), and cannot return to the depot before the specified time T ends;

[0144] The electric vehicle is currently in the depot, and has no remaining goods at any customer vertex, then all vertices except the depot are shielded;

[0145] S24. LSTM decoder: use LSTM to model the decoder network, as shown in Figure 3 During decoding, the LSTM receives the vector representation of the current position A of the EV and the memory state of the previous decoding step, and outputs a hidden state B, which retains its trajectory information, i.e. C;

[0146] S3. Use random sampling method to sample the solution of the graph structure, quickly and comprehensively explore the solution space;

[0147] Use random sampling method to thoroughly explore the solution space, for all vertices i, at each decoding step t, according to p i t The described probability distribution samples the next vertex to be visited, and this process is repeated to obtain multiple solutions for an instance, and the shortest solution is reported;

[0148] S4. Use rollout baseline policy gradient algorithm to train the attention model, get the trained attention model, which is used to find the shortest path for the electric vehicle fleet, as follows:

[0149] S41. Define the loss function

[0150] The loss function represents the negative expected reward of the sampled trajectory Y using the random policy θ , the formula is as follows:

[0151]

[0152] S42. Gradient estimation

[0153] Estimate the gradient of the loss function L(θ) with respect to the trainable variable θ, the formula is as follows:

[0154]

[0155] In the formula, X t is the information set; Gt is a global variable set; Y t denotes the sequence of vertices formed before step t; parameter N is the batch size, X [i] is the i-th training example in the batch, Y [i] is the solution generated using θ the corresponding solution generated; BL() denotes the rollout baseline, P θ (Y [i] | X [i] ) denotes the probability of generating a solution Y [i] using a random policy θ ; [i]

[0156] S43. Instance generation

[0157] In each training step, N time-windowed electric vehicle routing problem training instances are randomly generated, and the shortest path for the electric vehicle fleet is quickly and efficiently found, so that the total distance traveled by the fleet is minimized.

[0158] In summary, the present application can capture the structure embedded in the given graph, find the best path for the electric vehicle fleet under the condition of ensuring customer demand and time information, minimize the total distance traveled by the fleet, support super large-scale electric vehicle fleet operation instances, and balance quality and efficiency.

[0159] Embodiment two:

[0160] The embodiment provides a path optimization system for large-scale electric vehicle fleet operation, which is used for implementing the path optimization method for large-scale electric vehicle fleet operation described in embodiment one, and comprises:

[0161] A definition module is configured to define a time-windowed electric vehicle routing problem based on a graph structure, and describe a system state, a reward scheme and a shielding strategy of the time-windowed electric vehicle routing problem from the perspective of reinforcement learning.

[0162] A construction module is configured to construct an attention model, and combine the attention model with a graph embedding component, so as to aggregate local and global information of the graph structure.

[0163] A sampling module is configured to sample a solution result of the graph structure by using a random sampling method.

[0164] A training output module is configured to train the attention model by using a rollout baseline policy gradient algorithm, so as to obtain a trained attention model, which is used for finding the shortest path for the electric vehicle fleet.

[0165] ​Specifically, the definition module, the construction module, the sampling module and the training output module can be embedded into a computer processing system. The computer calls the modules to complete the task of electric vehicle fleet path optimization according to the above-mentioned method for path optimization of large-scale electric vehicle fleet operation. The definition module, the construction module, the sampling module and the training output module can perform operations according to the specific steps given by the method for path optimization of large-scale electric vehicle fleet operation.

[0166] It should be noted that the division of each module of the above system is only a logical function division. In actual implementation, all or part of the modules can be integrated into one physical entity, or can be physically separated, and the modules can all be implemented in the form of software called by a processing element; all can be implemented in the form of hardware; some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the definition module can be a separate processing element, or can be integrated into a chip of the above device. In addition, the definition module can also be stored in the form of program code in the memory of the above device, and the function of the above signal processing module can be called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, all or part of the modules can be integrated together or independently implemented. The processing element described herein can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of the hardware in the processor element or the instruction in the form of software.

[0167] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module is implemented in the form of program code called by a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, the modules can be integrated together to implement in the form of system-on-a-chip (SOC).

[0168] Embodiment three:

[0169] The application provides a terminal device, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the above-mentioned path optimization method for large-scale electric vehicle fleet operation is adopted.

[0170] It should be noted that the terminal device can adopt a computer device such as a desktop computer, a notebook computer or a cloud server, and the terminal device comprises but is not limited to a processor and a memory, for example, the terminal device can further comprise an input / output device, a network access device and a bus, etc.

[0171] Further, the processor can adopt a central processing unit (CPU), of course, according to the actual use, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), ready-to-program gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. can also be adopted, the general-purpose processor can adopt a microprocessor or any conventional processor, etc., and the present application does not limit this.

[0172] It should be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0173] For those skilled in the art, the specific meaning of the above-mentioned terms in the present application can be understood according to the specific circumstances. When an element is referred to as "assembled", "mounted", "fixed" or "disposed" on another element, it can be directly on another element or there can be a middle element. When an element is considered to be "connected" to another element, it can be directly connected to another element or there can be a middle element. The terms "vertical", "horizontal", "up", "down", "left", "right" and similar expressions used herein are only for illustrative purposes and are not the only embodiment.

[0174] While embodiments of the application have been shown and described, it is to be understood that the application is not limited to the details of the embodiments described, since numerous changes, modifications, substitutions, and alterations can be made thereto without departing from the spirit and scope of the application as defined in the appended claims and their equivalents.

[0175] In the description of the specification, reference to "one embodiment", "an example", "a specific example" or the like means that a particular feature, structure, material or characteristic is included in at least one embodiment or example of the disclosure. The appearances of the above expressions in various places in the specification do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

Claims

1. A route optimization method for large-scale electric vehicle fleet operations, characterized in that: The following steps are involved: The electric vehicle routing problem with time windows is defined based on a graph structure, and the system state, reward scheme, and shielding strategy of the electric vehicle routing problem with time windows are described from the perspective of reinforcement learning. Build an attention model and combine it with the graph embedding component to aggregate local and global information of the graph structure: The attention model consists of a graph embedding component, an attention mechanism, and an LSTM decoder; Graph Embedding Component First, the input information set X of the attention model t and the global variable G t Mapped to high-dimensional vector space, respectively recorded as and And use the Structure2Vec tool to synthesize the embedding vector; For each vertex vector i, the vector Initialize and then update using the recursive formula After p rounds of recursion, the attention network model will generate a ξ-dimensional vector for each vertex i And set for In each round of recursion, global information and position information are aggregated through the first two terms of the recursive formula, and information of different vertices and edges is propagated to each other through the last two summation terms, so that the final embedding vector It contains both local information and global information. The recursive formula is as follows: ∑ j∈N ( i )relu[θ5w(i,j)]}(6) Where N(i) is the set of vertices connected to vertex i by an edge, and this set is called the neighborhood of vertex i. w(i,j) represents the travel time on edge (i,j). θ1, θ2, θ3, θ4, and θ5 are trainable variables. Relu is a nonlinear activation function, and relu(x) = max{0,x}. Attention Mechanism Based on embedding vector Utilize the context-based attention mechanism to calculate the visit probability of each vertex i; First calculate the context vector c t , the state of the entire graph structure is specified as the weighted sum of all embedding vectors, that is: The weight of each vertex is defined as follows: a t =softmax(v t ) (8) Where a t context vector, is the vector v t The i-th item, h t is the hidden memory state of the LSTM decoder, θ v and θ u is a trainable variable, [;] means connecting the two vectors on both sides of the symbol ";", and tanh is a nonlinear activation function. is the normalized exponential function applied to the vector, Then, in The probability of visiting each vertex i is estimated in the following formula: p t =softmax(p t ) (10) Where p t context vector, is the vector g t The i-th entry, θ c and θ g is a trainable variable; Blocking strategy Design the following blocking strategies to exclude infeasible routes, as follows: Vertex j represents a customer whose unsatisfied demand is zero or exceeds the remaining goods of the electric vehicle; Vertex j represents the customer, and the current battery charge of the electric vehicle is b t It is impossible to support electric vehicles to complete the journey from vertex i to vertex j and then to the depot; The earliest time to reach vertex j violates the time window constraint, i.e., τ t +w(i,j)>l j ; The electric vehicle travels from vertex i to vertex j and cannot return to the warehouse before the planned time T ends; If the electric vehicle is currently at the warehouse and has no remaining goods on any customer vertex, all vertices except the warehouse are blocked; LSTM decoder: Use LSTM to model the decoder network; The random sampling method is used to sample the graph structure solution results; The rollout baseline policy gradient algorithm is used to train the attention model, and the trained attention model is used to find the shortest path for the electric vehicle fleet.

2. The path optimization method for large-scale electric vehicle fleet operation according to claim 1, characterized in that: The electric vehicle routing problem with time windows is defined based on the graph structure as follows: (21) The weight of each edge of the graph is defined as the Euclidean distance between the connected vertices. The graph structure includes three types of vertices: Client V c 、Station V s and Warehouse V d ; At this time, the solution to the electric vehicle routing problem with a time window is the vertex in the graph structure, which is the planned route of the electric vehicle; (22) Each vertex i is associated with an array Associated, all vertex arrays form An information set X t , used to decode the local information at the vertex at step t, where x i and z i represents the geographic coordinates of vertex i, e i and l i represents the corresponding time window, represents the remaining demand of vertex i at decoding step t; (23) Each vertex shares a set of global variables: G t ={τ t ,b t ,s t } Where τ t represents the time of the active EV at the beginning of decoding step t, with the initial value set to 0; b t Represents the battery power, and the initial value is set to the battery capacity of the electric vehicle; σ t represents the number of available electric vehicles, and its initial value is set to the size of the fleet.

3. The path optimization method for large-scale electric vehicle fleet operation according to claim 2, characterized in that: Based on the reinforcement learning perspective, the system state, reward scheme, and shielding strategy of the electric vehicle routing problem with time windows are described as follows: (31) In the electric vehicle routing problem with time windows, the system state is the information set X in the graph structure. t and the global variable G t the expression; Assume that there is an agent that adds a vertex to the end of the current sequence at each step based on the current system state and the given information. The system state will change accordingly, and this process will be repeated until the termination condition is met. Assume that at step t m Terminate at y, and set the termination condition to meet all customer requirements, define y t Denotes the vertex selected in step t, and defines Y t represents the vertex sequence formed before step t; At each decoding step t, given the set X t , global variable G t and travel history Y t , estimate the probability of each vertex i being added to the sequence through the probability distribution formula, and decode the next vertex y to be visited according to this probability distribution t+1 , the probability distribution formula is as follows: P(y t+1 =i|X t ,G t ,Y t ) Where, X t is the information set; G t is a set of global variables; Y t represents the vertex sequence formed before step t; (32) Based on the obtained y t+1 , use the transfer function formulas (1)-(4) to update the system state, as follows: (32.1) System time τ t+1 : Where V c Indicates customer, V s Indicates station, V d Indicates warehouse; Indicates taking the maximum value of the system time and the vertex; y t represents the vertex selected in step t, w(y t ,y t+1 ) is from vertex y t To vertex y t+1 travel time; b t Indicates battery power, re(b t ) is the time required to fully charge the battery from a given standard, and s is a constant representing the service time of each customer vertex; (32.2)Battery capacity b t+1 : Where b t Indicates battery capacity, V c represents the customer, y t represents the vertex selected at step t, f(y t ,y t+1 ) is the vertex y of the electric vehicle t To vertex y t+1 The energy consumption is, B is the battery capacity; (32.3) The number of electric vehicles available at each vertex is σ t and remaining demand renew: Where V d Represents the warehouse, y t represents the vertex selected at step t; (32.4) Set the reward function. The reward function is used to guide the attention model to generate solutions subject to conditional constraints. The first term in the reward function is set to the negative total distance traveled by the team to favor short-distance solutions. The other terms are penalties for the problem constraints, as follows: Where, σ t represents the number of electric vehicles, b t Indicates battery level, y t represents the vertex selected at step t, Y t represents the vertex sequence formed before step t, w(y t-1 ,y t ) is along the edge (y t-1 ,y t ) travel time, Along trajectory Y t m The number of site visits, β1, β2 and β3 are three negative constants.

4. The path optimization method for large-scale electric vehicle fleet operation according to claim 3, characterized in that: The random sampling method is used to sample the graph structure solution results, as follows: For all vertices i, at each decoding step t, according to The described probability distribution samples the next vertex to be visited, repeats this process to obtain multiple solutions for an instance, and reports the solution with the shortest distance.

5. The path optimization method for large-scale electric vehicle fleet operation according to claim 4, characterized in that: The rollout baseline policy gradient algorithm is used to train the attention model. The trained attention model is used to find the shortest path for the electric vehicle fleet as follows: (51) Define the loss function The loss function represents the use of a random policy π θ The negative expected reward of the sampled trajectory Y is as follows: (52) Gradient estimation Estimate the gradient of the loss function L(θ) with respect to the trainable variable θ as follows: Where, X t is the information set; G t is a set of global variables; Y t Represents the vertex sequence formed before step t; parameter N is the batch size, X [i] is the i-th training example in the batch, Y [i] To use π θ The corresponding solution generated; BL() represents the rollout baseline, P θ (Y [i] |X [i] ) represents a given training example X [i] Use a random policy π θ Generate solution Y [i] probability.

6. A path optimization system for large-scale electric vehicle fleet operations, used to implement the path optimization method for large-scale electric vehicle fleet operations according to any one of claims 1 to 5, characterized in that: include: A definition module is used to define the electric vehicle routing problem with time windows based on a graph structure, and to describe the system state, reward scheme, and shielding strategy of the electric vehicle routing problem with time windows from the perspective of reinforcement learning; A building module for building an attention model, combining the attention model with the graph embedding component to aggregate local and global information of the graph structure; Sampling module, used to sample the graph structure solution results using random sampling method; The training output module is used to train the attention model using the rollout baseline policy gradient algorithm to obtain the trained attention model, which is used to find the shortest path for the electric vehicle fleet.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the path optimization method for large-scale electric vehicle fleet operation according to any one of claims 1 to 5 is adopted.

Images