Path decision method combining decision transformer and gpg optimization just-in-time arrival probability

Abstract

The application discloses a path decision method combining Decision Transformer and GPG to optimize the probability of on-time arrival. The method comprises the following steps: obtaining a vehicle initial path data set; constructing a vehicle path decision model based on Decision Transformer; obtaining the probability distribution of vehicle path node selection by using the vehicle initial path data set and the vehicle path decision model to generate a vehicle predicted path; calculating the policy gradient based on the vehicle predicted path and GPG with the probability of on-time arrival as the optimization target; performing back propagation training on the vehicle path decision model by using the policy gradient; and performing vehicle path decision by using the policy gradient. The application can solve the problem of insufficient spatial correlation modeling in the prior art, improve the accuracy and stability of path decision, reduce the dependence on prior knowledge in the optimization decision process, and thus realize the optimization of the probability of on-time arrival in path decision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a path decision method for optimizing punctual arrival probability combining Decision Transformer and GPG (Generalized Policy Gradient). BACKGROUND

[0002] Reliable navigation in stochastic traffic networks has always been the cornerstone of intelligent transportation system services. As of now, among numerous reliable navigation objectives, optimizing the punctual arrival probability of vehicles has become a widely applied key objective, especially for various time-sensitive navigation scenarios.

[0003] Traditional path decision methods can be divided into data-driven and model-driven methods. Data-driven methods usually assume that path travel times follow a Markov process, i.e., future path travel times are not affected by the past. However, this assumption is difficult to hold in practical applications, resulting in poor adaptability of the model to complex road networks. Model-driven methods usually require assumptions about the distribution of path travel times and rely on strong prior knowledge, making it difficult to accurately describe the true distribution of path times and lacking in spatial correlation modeling. Moreover, both data-driven and model-driven methods fail to effectively integrate the spatiotemporal features of path dynamics, resulting in low punctual arrival probability of path decision. SUMMARY

[0004] In view of the above deficiencies in the prior art, the present application provides a path decision method for optimizing punctual arrival probability combining Decision Transformer and GPG.

[0005] To achieve the above-mentioned application purposes, the technical solution adopted by the present application is as follows:

[0006] The path decision method for optimizing punctual arrival probability combining Decision Transformer and GPG comprises the following steps:

[0007] S1, obtaining a vehicle initial path data set based on a road network;

[0008] S2, constructing a vehicle path decision model based on Decision Transformer, obtaining a probability distribution of vehicle path node selection using the vehicle initial path data set and the vehicle path decision model, and generating a vehicle predicted path according to the probability distribution of vehicle path node selection;

[0009] S3, calculating a policy gradient with punctual arrival probability as the optimization objective based on the vehicle predicted path and GPG, performing backpropagation training of the vehicle path decision model using the policy gradient, and performing vehicle path decision using the vehicle path decision model after backpropagation training.

[0010] Furthermore, in step S1, the vehicle initial path dataset includes vehicle path node data, vehicle node passage time data, vehicle remaining time budget data, and vehicle current node data.

[0011] Further, in step S2, the vehicle routing decision model includes an attention embedding layer, a linear embedding layer, an Encoder, and a Decoder; the input of the attention embedding layer serves as the first data input, the output of the attention embedding layer is connected to the input of the Encoder, the output of the Encoder is connected to the first input of the Decoder, the input of the linear embedding layer serves as the second data input, the output of the linear embedding layer is connected to the second input of the Decoder, and the output of the Decoder serves as the output of the vehicle routing decision model; the data input to the first data input is the vehicle's path through node data and the vehicle's node passage time data; the data input to the second data input is the vehicle's remaining time budget data and the vehicle's current node data.

[0012] Furthermore, the expression for data processing in the attention embedding layer is:

[0013] P′=EmbeddingP(P),

[0014] P = [p1, p2, ..., p L ],

[0015] R′=W R R+b R ,

[0016] R = [r1, r2, ..., r L ],

[0017] X fusion =Attention(P′,R′,R′)

[0018] Where: P′ represents the embedded data of nodes traversed by the vehicle path, EmbeddingP is the lookup table of nodes traversed by the vehicle path, P represents the data of nodes traversed by the vehicle path, p1 is the first node traversed by the vehicle path, p2 is the second node traversed by the vehicle path, p... L Let R' be the Lth node passed by the vehicle along its path, where L is the number of nodes passed by the vehicle along its path, R′ is the embedded data of the vehicle node's passage time, and W is the number of nodes passed by the vehicle along its path. R Here, b represents the weighting coefficient for vehicle node transit time, R represents vehicle node transit time data, and b represents the weighting coefficient. R Let r1 be the vehicle's time-of-travel bias coefficient at node 1, r2 be the vehicle's time-of-travel at node 2, and r3 be the time-of-travel bias coefficient at node 3. L Let X be the time it takes for the vehicle to pass through the Lth node.fusion X is the fusion feature of the vehicle, and Attention is the standard self-attention mechanism.

[0019] Further, the Encoder includes 4 layers of TransformerBlock, and the expression of data processing in the first layer of TransformerBlock of the Encoder is:

[0020] X out1 = TransformerBlock(X in1 + R′),

[0021] X in1 = Dropout(X fusion ) + Embedding(P′)

[0022] wherein X out1 is the output of the first layer of TransformerBlock of the Encoder, X in1 is the input of the first layer of TransformerBlock of the Encoder, R′ is the time embedding data of the vehicle node, Dropout is the regularization processing, X fusion is the fusion feature of the vehicle, Embedding is the position encoding processing, and P′ is the node embedding data of the vehicle path;

[0023] The output of the first layer of TransformerBlock of the Encoder is taken as the input of the second layer of TransformerBlock of the Encoder, and the above processing process is sequentially performed, and the output X out4 of the fourth layer of TransformerBlock of the Encoder is taken as the output X enc of the Encoder.

[0024] Further, the expression of data processing in the linear embedding layer is:

[0025] p s ′ = EmbeddingS(p s ),

[0026] B′ = W B B + b B

[0027] wherein p s ′ is the current node embedding data of the vehicle, EmbeddingS is the lookup table of the current node of the vehicle, p s is the current node data of the vehicle, B′ is the remaining time budget embedding data of the vehicle, W BB represents the weighting factor for the vehicle's remaining time budget, and B represents the vehicle's remaining time budget data. B The bias coefficient is the budget for the remaining time of the vehicle.

[0028] Furthermore, the Decoder includes four TransformerBlock layers and a prediction module;

[0029] The expression for data processing in the first-level TransformerBlock of the Decoder is:

[0030] X out1 = TransformerBlock(X) in1 ′,X enc ,X enc ),

[0031] X in1 = Dropout(B'+p) s ′)

[0032] Where: X out1 ' is the output of the first layer of the Decoder's TransformerBlock, X enc X is the output of the Encoder. in1 ' is the input to the first layer of the Decoder's TransformerBlock, Dropout is used for regularization, and p s B' represents the embedded data of the current node of the vehicle, and B' represents the embedded data of the vehicle's remaining time budget.

[0033] The output of the first TransformerBlock of the Decoder is used as the input of the second TransformerBlock of the Decoder, and the above processing is performed sequentially. The output X of the fourth TransformerBlock of the Decoder... out4 As input to the Decoder's prediction module;

[0034] The expression for data processing in the Decoder's prediction module is:

[0035] y = softmax(fc(X) out4 ′))

[0036] Where: y is the probability of vehicle path node selection, softmax is the softmax activation function, fc is the fully connected layer, and X... out4 ′ represents the output of the fourth layer TransformerBlock of the Decoder.

[0037] Further, in step S3, based on the vehicle predicted path and the GPG, a strategy gradient is calculated with the punctual arrival probability as the optimization target, including the following steps:

[0038] A1, calculating the required duration of the vehicle predicted path;

[0039] A2, constructing a target function of the vehicle path decision model with the punctual arrival probability as the optimization target based on the required duration of the vehicle predicted path;

[0040] A3, constructing an initial strategy gradient model based on the target function of the vehicle path decision model and the GPG, updating the initial strategy gradient model by using the Monte Carlo sampling method and the fixed numerical baseline method to obtain a final strategy gradient model, and calculating the strategy gradient by using the final strategy gradient model.

[0041] Further, in step A3, the expression of the final strategy gradient model is:

[0042]

[0043] wherein: is the strategy gradient, M is the number of vehicle predicted paths generated by the vehicle path decision model through the Monte Carlo sampling method, j is the number of the predicted path generated by the vehicle path decision model through the Monte Carlo sampling method, is an indicator function of the condition R(τ (j) )≤T, indicating that when the condition R(τ (j) )≤T is true, the value is 1, and when the condition R(τ (j) )≤T is false, the value is 0, R(τ (j) ) is the required duration of the vehicle predicted path τ (j) , τ (j) is the jth vehicle predicted path generated by the vehicle path decision model through the Monte Carlo sampling method, T is the target time, b is the baseline value, k is the node number in the vehicle predicted path τ (j) , H j is the number of nodes in the vehicle predicted path τ (j) , is the logarithmic strategy gradient, is the strategy corresponding to the action executed when the current state of the vehicle is , is the action corresponding to the selection of the k+1th node at the kth node in the jth vehicle predicted path, is the state at the kth node in the jth vehicle predicted path.

[0044] Further, the expression of the node state in the vehicle predicted path is:

[0045] s k = (p s，k ', y k+1 , B k ', S k ),

[0046] S k = (p k-1 ', r k ', S k-1 , a k-1 )

[0047] wherein: s k is a state at the kth node in the vehicle prediction path, p s，k ' is the current node embedding data of the vehicle at the kth node in the vehicle prediction path, y k+1 is the probability of the vehicle prediction path selecting the k+1th node at the kth node, B k ' is the remaining time budget embedding data of the vehicle at the kth node in the vehicle prediction path, S k is the historical data of the vehicle at the kth node in the vehicle prediction path, p k-1 ' is the k-1th node embedding data of the vehicle prediction path, r k ' is the passing time embedding data of the vehicle at the kth node in the vehicle prediction path, S k-1 is the historical data of the vehicle at the k-1th node in the vehicle prediction path, and a k-1 is the execution action corresponding to the selection of the kth node at the k-1th node in the vehicle prediction path.

[0048] The present application has the following beneficial effects:

[0049] (1) The present application can effectively model the dynamic changes of path time by using the ability of Decision Transformer to handle long-term time dependence, solve the problem of insufficient spatial correlation modeling in the prior art, and further improve the accuracy and stability of path decision making;

[0050] (2) The present application can reduce the dependence on prior knowledge by calculating the strategy gradient with the on-time arrival probability as the optimization target based on the vehicle prediction path and GPG, using the strategy gradient to train the vehicle path decision model through back propagation, and using the vehicle path decision model trained through back propagation to make vehicle path decisions, thereby achieving the on-time arrival probability of optimizing path decision making;

[0051] (3) This invention updates the initial policy gradient model by using Monte Carlo sampling and fixed numerical baseline methods, which can reduce the variance of policy updates during reinforcement learning and improve training convergence. Attached Figure Description

[0052] Figure 1 A schematic diagram of the path decision-making method that combines Decision Transformer and GPG to optimize the probability of on-time arrival;

[0053] Figure 2 This is a schematic diagram of the data processing and backpropagation training process of the path decision model of the present invention. Detailed Implementation

[0054] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0055] like Figure 1 As shown, the path decision-making method that combines Decision Transformer and GPG to optimize the probability of on-time arrival includes steps S1-S3, as detailed below:

[0056] S1. Obtain the initial vehicle path dataset based on the road network.

[0057] In an optional embodiment of the present invention, the vehicle initial path dataset includes vehicle path node data, vehicle node passage time data, vehicle remaining time budget data, and vehicle current node data.

[0058] The road network in this invention is a given road network with spatial correlation. This invention uses Monte Carlo sampling to simulate the road network to obtain the initial vehicle path dataset. By utilizing Monte Carlo sampling to simulate the road network and obtain the initial vehicle path dataset, this invention avoids making distributional assumptions about vehicle node transit times, i.e., it eliminates the need for a mathematical model of the path transit time distribution, thereby improving the adaptability of this invention under different road network structures.

[0059] S2. Construct a vehicle path decision model based on Decision Transformer, use the initial vehicle path dataset and the vehicle path decision model to obtain the probability distribution of vehicle path node selection, and generate the vehicle predicted path based on the probability distribution of vehicle path node selection.

[0060] In an optional embodiment of the present application, the vehicle path decision model comprises an attention embedding layer, a linear embedding layer, an Encoder and a Decoder, as shown in Figure 2 ; the input end of the attention embedding layer serves as a first data input end, the output end of the attention embedding layer is connected to the input end of the Encoder, the output end of the Encoder is connected to the first input end of the Decoder, the input end of the linear embedding layer serves as a second data input end, the output end of the linear embedding layer is connected to the second input end of the Decoder, and the output end of the Decoder serves as the output of the vehicle path decision model; the data input by the first data input end is vehicle path passing node data and vehicle node passing time data; and the data input by the second data input end is vehicle remaining time budget data and vehicle current node data.

[0061] The attention embedding layer is configured to convert the vehicle path passing node data into vehicle path passing node embedding data through a lookup table, convert the vehicle node passing time data into vehicle node passing time embedding data through linear transformation, and fuse the vehicle path passing node embedding data and the vehicle node passing time embedding data through an attention mechanism to obtain the fusion feature of the vehicle; the Encoder is configured to perform road network feature extraction on the fusion feature of the vehicle to output deep features of road network spatial correlation; the linear embedding layer is configured to convert the vehicle current node data into vehicle current node embedding data through a lookup table and convert the vehicle remaining time budget data into vehicle remaining time budget embedding data through linear transformation; and the Decoder is configured to perform data processing on the input deep features of road network spatial correlation, the vehicle current node embedding data and the vehicle remaining time budget embedding data through a self-attention mechanism to obtain the probability distribution of vehicle path node selection.

[0062] The expression of data processing in the attention embedding layer is:

[0063] P′=EmbeddingP(P),

[0064] P=[p1,p2,...,p L ],

[0065] R′=W R R+b R ,

[0066] R=[r1,r2,...,r L ],

[0067] X fusion =Attention(P′,R′,R′)

[0068] wherein: P' is the vehicle path passing node embedding data, EmbeddingP is the vehicle path passing node lookup table, P is the vehicle path passing node data, p1 is the first node passed by the vehicle path, p2 is the second node passed by the vehicle path, p L is the Lth node passed by the vehicle path, L is the number of vehicle path passing nodes, R' is the vehicle node passing time embedding data, W R is the weight coefficient of the vehicle node passing time, R is the vehicle node passing time data, b R is the bias coefficient of the vehicle node passing time, r1 is the passing time of the vehicle at the first node, r2 is the passing time of the vehicle at the second node, r L is the passing time of the vehicle at the Lth node, X fusion is the fusion feature of the vehicle, Attention is the standard self-attention mechanism, which can capture the correlation information between the path and the time to obtain the fusion feature, wherein the attention head dimension is 8 and the number of layers is 2;

[0069] The Encoder includes 4 layers of TransformerBlock, and the expression of data processing in the first layer of TransformerBlock of the Encoder is:

[0070] X out1 = TransformerBlock(X in1 + R'),

[0071] X in1 = Dropout(X fusion ) + Embedding(P')

[0072] wherein: X out1 is the output of the first layer of TransformerBlock of the Encoder, TransformerBlock includes a self-attention mechanism and FFN (Feed Forward Network), and uses residual connection and layer normalization to stabilize training, X in1 is the input of the first layer of TransformerBlock of the Encoder, R' is the vehicle node passing time embedding data, Dropout is a regularization process, X fusion is the fusion feature of the vehicle, Embedding is a position encoding process, and P' is the vehicle path passing node embedding data.

[0073] The output of the first layer of the TransformerBlock of the Encoder is taken as the input of the second layer of the TransformerBlock of the Encoder, and the above processing process is sequentially performed, and the output X of the fourth layer of the TransformerBlock of the Encoder out4 As the output X of the Encoder enc , X enc is a deep feature related to the spatial correlation of the road network.

[0074] The expression of data processing in the linear embedding layer is:

[0075] p s ′=EmbeddingS(p s ),

[0076] B′=W B B+b B

[0077] Where: p s ′ is the current node embedding data of the vehicle, EmbeddingS is the current node lookup table of the vehicle, p s is the current node data of the vehicle, B′ is the remaining time budget embedding data of the vehicle, W B is the weight coefficient of the remaining time budget of the vehicle, B is the remaining time budget data of the vehicle, and b B is the bias coefficient of the remaining time budget of the vehicle.

[0078] The Decoder includes 4 layers of TransformerBlock and a prediction module.

[0079] The expression of data processing in the first layer of the TransformerBlock of the Decoder is:

[0080] X out1 ′=TransformerBlock(X in1 ′,X enc ,X enc ),

[0081] X in1 ′=Dropout(B′+p s ′)

[0082] Where: X out1 ′ is the output of the first layer of the TransformerBlock of the Decoder, the TransformerBlock includes a self-attention mechanism and an FFN (Feed Forward Network), and uses a residual connection and layer normalization to stabilize training, X encThe output of the Encoder, X in1 The input of the first layer of the Decoder, Dropout is a regularization process, p s The current node embedding data of the vehicle, B' is the remaining time budget embedding data of the vehicle.

[0083] The output of the first layer of the Decoder is the input of the second layer of the Decoder, and the above process is sequentially performed, and the output of the fourth layer of the Decoder X out4 The input of the prediction module of the Decoder.

[0084] The expression of data processing in the prediction module of the Decoder is:

[0085] y = softmax(fc(X out4 '))

[0086] Wherein: y is the probability of vehicle path node selection, softmax is a softmax activation function, fc is a full connection layer, X out4 ' is the output of the fourth layer of the Decoder.

[0087] S3, based on the vehicle predicted path and the GPG, the strategy gradient is calculated with the on-time arrival probability as the optimization target, the vehicle path decision model is trained by back propagation using the strategy gradient, and the vehicle path decision is made using the vehicle path decision model trained by back propagation.

[0088] In an optional embodiment of the present application, the strategy gradient is calculated based on the vehicle predicted path and the GPG with the on-time arrival probability as the optimization target, comprising the following steps:

[0089] A1, based on the vehicle predicted path, the time required is calculated.

[0090] A2, based on the time required by the vehicle predicted path, a target function of the vehicle path decision model is constructed with the on-time arrival probability as the optimization target.

[0091] The expression of the target function of the vehicle path decision model is:

[0092] J s (π θ ) = P[R(τ) ≤ T]

[0093] Wherein: J s (π θ ) is the path decision model π θThe objective function of the vehicle path decision model, and θ is a weight parameter of the vehicle path decision model, θ=(W R , b R , W B , b B ), W R is a weight coefficient of the vehicle node passing time, b R is a bias coefficient of the vehicle node passing time, W B is a weight coefficient of the vehicle remaining time budget, b B is a bias coefficient of the vehicle remaining time budget, P[R(τ)≤T] is a probability that the required time R(τ) of the vehicle predicted path is less than or equal to the target time T, and τ is the vehicle predicted path.

[0094] A3, based on the objective function of the vehicle path decision model and the GPG, an initial policy gradient model is constructed, and the initial policy gradient model is updated by using the Monte Carlo sampling method and the fixed numerical baseline method to obtain a final policy gradient model, and the final policy gradient model is used to calculate the policy gradient.

[0095] The expression of the initial policy gradient model is:

[0096]

[0097] The expression of the initial policy gradient model updated by the Monte Carlo sampling method is:

[0098]

[0099] The expression of the final policy gradient model is:

[0100]

[0101] Among them: is the policy gradient, M is the number of vehicle predicted paths generated by the vehicle path decision model through the Monte Carlo sampling method, j is the predicted path number generated by the vehicle path decision model through the Monte Carlo sampling method, is an indicator function of the condition R(τ (j) )<T, which takes a value of 1 when the condition R(τ (j) )<T is true, and takes a value of 0 when the condition R(τ (j) )<T is false, R(τ (j) ) is the required time of the vehicle predicted path τ (j) , τ (j) is the jth vehicle predicted path generated by the vehicle path decision model through the Monte Carlo sampling method, T is the target time, b is the baseline value, k is the node number in the vehicle predicted path τ (j) , and H j is the vehicle predicted path τ(j) The number of nodes in For the logarithmic policy gradient, The current status of the vehicle is Execute action at time The corresponding strategy, To select the action corresponding to the (k+1)th node at the kth node in the j-th vehicle prediction path, Let be the state at the k-th node in the j-th vehicle prediction path.

[0102] The expression for the node state in the vehicle prediction path is:

[0103] s k =(p s，k ′,y k+1 B k ′, S k ),

[0104] S k =(p k-1 ′, r k ′,S k-1 ,a k-1 )

[0105] Where: s k Let p be the state at the k-th node in the vehicle's predicted path. s，k ' represents the embedded data of the current node of the vehicle at the k-th node in the vehicle prediction path, y k+1 Let B be the probability that a vehicle will choose the (k+1)th node at the k-th node in its predicted path. k S' is the vehicle remaining time budget embedding data at the k-th node in the vehicle prediction path. k For the historical vehicle data at the k-th node in the vehicle prediction path, p k-1 ′ represents the embedded data of the (k-1)th node traversed by the vehicle's predicted path, r k S' represents the embedded data of the vehicle's transit time at the k-th node in the vehicle's predicted path. k-1 For the historical vehicle data at the (k-1)th node in the vehicle prediction path, a k-1 Select the action corresponding to the kth node for the (k-1)th node in the vehicle's predicted path.

[0106] The expression for backpropagation training of the vehicle path decision model using policy gradients in this invention is as follows:

[0107]

[0108] Where: θ′ are the weight parameters of the path decision model after backpropagation training, and α is the learning rate.

[0109] The present application is described in reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device that implements the flow Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks

[0110] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices that implement the flow Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks

[0111] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the flow Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks

[0112] The principles and implementation of the present application are described in the specific embodiments, and the above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed, and the above description should not be understood as a limitation of the present application.

[0113] Those skilled in the art will realize that the embodiments described herein are for the purpose of helping the reader to understand the principles of the present application, and should be understood as not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. A path decision method for optimizing the probability of on-time arrival by combining Decision Transformer and GPG, characterized in that, The method comprises the following steps: S1, obtaining a vehicle initial path data set based on a road network; S2, constructing a vehicle path decision model based on a Decision Transformer, obtaining a probability distribution of vehicle path node selection by using the vehicle initial path data set and the vehicle path decision model, and generating a vehicle predicted path according to the probability distribution of vehicle path node selection; S3, calculating a strategy gradient based on the vehicle predicted path and the GPG with the on-time arrival probability as an optimization target, performing back propagation training on the vehicle path decision model by using the strategy gradient, and performing vehicle path decision by using the vehicle path decision model after the back propagation training; The method for calculating the strategy gradient based on the vehicle predicted path and the GPG with the on-time arrival probability as the optimization target comprises the following steps: A1, calculating a required time length of the vehicle predicted path; A2, constructing an objective function of the vehicle path decision model with the on-time arrival probability as the optimization target based on the required time length of the vehicle predicted path; A3, constructing an initial strategy gradient model based on the objective function of the vehicle path decision model and the GPG, updating the initial strategy gradient model by using a Monte Carlo sampling method and a fixed numerical baseline method to obtain a final strategy gradient model, and calculating the strategy gradient by using the final strategy gradient model; The expression of the final strategy gradient model is: in: Let M be the policy gradient, M be the number of vehicle prediction paths generated by the vehicle routing decision model using the Monte Carlo sampling method, and j be the prediction path number generated by the vehicle routing decision model using the Monte Carlo sampling method. As a condition The indicator function indicates that when the condition is met... The value is 1 when the condition is true. When false, the value is 0. Predicting routes for vehicles The required time, This is the j-th predicted vehicle path generated by the vehicle routing decision model using the Monte Carlo sampling method, where T is the target time, b is the baseline value, and k is the predicted vehicle path. The node number in H j Predicting routes for vehicles The number of nodes in For the logarithmic policy gradient, The current status of the vehicle is Execute action at time The corresponding strategy, To select the action corresponding to the (k+1)th node at the kth node in the j-th vehicle prediction path, Let be the state at the k-th node in the j-th vehicle prediction path.

2. The path decision method of claim 1, wherein, In step S1, the vehicle initial path data set comprises vehicle path passing node data, vehicle node passing time data, vehicle remaining time budget data and vehicle current node data.

3. The path decision method of claim 1, wherein, In step S2, the vehicle path decision model comprises an attention embedding layer, a linear embedding layer, an Encoder and a Decoder; the input end of the attention embedding layer is a first data input end, the output end of the attention embedding layer is connected to the input end of the Encoder, the output end of the Encoder is connected to the first input end of the Decoder, the input end of the linear embedding layer is a second data input end, the output end of the linear embedding layer is connected to the second input end of the Decoder, and the output end of the Decoder is the output of the vehicle path decision model; the data input to the first data input end is the vehicle path passing node data and the vehicle node passing time data; and the data input to the second data input end is the vehicle remaining time budget data and the vehicle current node data.

4. The path decision method of claim 3, wherein, The expression of data processing in the attention embedding layer is: P′=EmbeddingP(P), P = [p1, p2,..., p L ], R' = W R R + b R , R=[r1,r2,...,r L ]， X fusion = Attention(P', R', R') wherein: P' is the vehicle path passing node embedding data, EmbeddingP is the vehicle path passing node lookup table, P is the vehicle path passing node data, p1 is the first node passed by the vehicle path, p2 is the second node passed by the vehicle path, p L is the Lth node passed by the vehicle path, L is the number of vehicle path passing nodes, R' is the vehicle node passing time embedding data, W R is the weight coefficient of the vehicle node passing time, R is the vehicle node passing time data, b R is the bias coefficient of the vehicle node passing time, r1 is the passing time of the vehicle at the first node, r2 is the passing time of the vehicle at the second node, r L is the passing time of the vehicle at the Lth node, X fusion is the fusion feature of the vehicle, and Attention is a standard self-attention mechanism.

5. The path decision method of claim 3, wherein, The Encoder comprises four layers of TransformerBlock, and the expression of data processing in the first layer of TransformerBlock of the Encoder is: X out1 = TransformerBlock(X in1 + R'), X in1 = Dropout(X fusion ) + Embedding(P') wherein: X out1 is the output of the first layer of the Encoder, X in1 is the input of the first layer of the Encoder, R' is the time embedding data of the vehicle node, Dropout is the regularization processing, X fusion is the fusion feature of the vehicle, Embedding is the position encoding processing, and P' is the node embedding data of the vehicle path. The output of the first layer of the TransformerBlock of the Encoder is taken as the input of the second layer of the TransformerBlock of the Encoder, the output of the second layer of the TransformerBlock of the Encoder is taken as the input of the third layer of the TransformerBlock of the Encoder, the output of the third layer of the TransformerBlock of the Encoder is taken as the input of the fourth layer of the TransformerBlock of the Encoder, and the output X of the fourth layer of the TransformerBlock of the Encoder out4 as the output of the Encoder enc .

6. The path decision method of claim 3, wherein, The expression of data processing in the linear embedding layer is: p s ′=EmbeddingS(p s ), B' = W B B + b B where: p s is the vehicle current node embedding data, EmbeddingS is the vehicle current node lookup table, p s is the vehicle current node data, B' is the vehicle remaining time budget embedding data, W B is the weight coefficient of the vehicle remaining time budget, B is the vehicle remaining time budget data, b B is the bias coefficient of the vehicle remaining time budget.

7. The path decision method of claim 3, wherein, The Decoder comprises four layers of TransformerBlock and a prediction module. The expression of data processing in the first layer of TransformerBlock of the Decoder is: X out1 ′ = TransformerBlock(X in1 ′, X enc , X enc ), X in1 = Dropout(B' + p s ') wherein: X out1 is the output of the first layer of TransformerBlock of Decoder, X enc is the output of Encoder, X in1 is the input of the first layer of TransformerBlock of Decoder, Dropout is a regularization process, p s is the current node embedding data of the vehicle, B' is the remaining time budget embedding data of the vehicle; The output of the first layer TransformerBlock of the Decoder is taken as the input of the second layer TransformerBlock of the Decoder, the output of the second layer TransformerBlock of the Decoder is taken as the input of the third layer TransformerBlock of the Decoder, the output of the third layer TransformerBlock of the Decoder is taken as the input of the fourth layer TransformerBlock of the Decoder, and the output X of the fourth layer TransformerBlock of the Decoder out4 as the input of the prediction module of the Decoder; The expression of data processing in the prediction module of the Decoder is: y = softmax(fc(X out4 )) where: y is the probability of the vehicle path node selection, softmax is the softmax activation function, fc is the fully connected layer, X out4 is the output of the fourth layer of the Decoder’s TransformerBlock.

8. The path decision method of claim 1, wherein, The expression of the node state in the vehicle predicted path is: s k = ( p s，k′ , y k+1 , B k′ , S k ), S k = (p k-1 ′,r k ′,S k-1 ,a k-1） wherein: s k is a state at a kth node in a predicted path of the vehicle, p s，k′ is vehicle current node embedding data at the kth node in the predicted path of the vehicle, yk+1 is a probability of the vehicle predicting a k+1th node at the kth node in the predicted path, B k′ is vehicle remaining time budget embedding data at the kth node in the predicted path of the vehicle, S k is vehicle historical data at the kth node in the predicted path of the vehicle, p k-1 is k-1th node embedding data of the predicted path of the vehicle, r k is a passing time embedding data of the vehicle at the kth node in the predicted path of the vehicle, S k-1 is vehicle historical data at the k-1th node in the predicted path of the vehicle, a k-1 is an execution action corresponding to selecting the kth node at the k-1th node in the predicted path of the vehicle.

Images