Space-time behavior prediction method based on multi-view personalized inverse reinforcement learning

Abstract

The invention discloses a spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning, and relates to the field of spatiotemporal prediction. The method comprises the following steps: extracting spatial multiple views from original deliveryman spatio-temporal behavior data; spatial multi-views are input into a personalized adversarial reinforcement learning network, the network comprises a generator, a discriminator and a value estimator, the generator comprises a multi-view encoder and a decoder, the input passes through the multi-view encoder to obtain region and point position representation, and the decoder outputs time-space behaviors of deliverymen; the discriminator takes the predicted and real time-space behaviors as input for training; the value estimator calculates an expected return of the current state; and the generator, the discriminator and the value estimator are alternately trained. According to the method, multiple views and personalized behavior preferences are introduced to the adversarial reinforcement learning network, the reward function is learned from the historical behavior data, the spatio-temporal behavior of the deliveryman is predicted, the interpretability of the model is enhanced, and the accuracy of spatio-temporal behavior prediction is improved.

Description

Technical Field

[0001] The present invention relates to the field of path planning, and in particular to a spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning. Background Art

[0002] In recent years, instant delivery services (such as express delivery and food delivery) have flourished and have become an important link between production and consumption. In the delivery service scenario, accurately predicting the spatiotemporal behavior of delivery personnel is a key link in realizing intelligent delivery. The spatiotemporal behavior prediction problem of delivery personnel involves very complex spatiotemporal factors, which is fundamentally different from the traditional combinatorial optimization problem of finding the minimum predefined time cost or distance cost and the prediction problem of selecting the next location from the location vocabulary. Most existing spatiotemporal behavior prediction technologies can only model simple spatiotemporal features, ignoring the influence of the delivery personnel's own personalized behavior preferences, resulting in a significant deviation between the actual spatiotemporal behavior of delivery personnel and the predicted spatiotemporal behavior, which makes it difficult to meet the spatiotemporal behavior prediction accuracy required for intelligent delivery applications.

[0003] Many existing delivery personnel spatiotemporal behavior prediction technologies, such as the DeepRoute method, which uses a Transformer-like encoder and an attention-based recurrent decoder, the RankETPA method, which uses recurrent neural network prediction, and the Graph2Route method, which proposes prediction based on dynamic spatiotemporal graphs. The drawback of these methods is that the test criteria are not differentiable, and the alternative training criteria used during model training are different from the test criteria, resulting in poor model test performance. Although the reinforcement learning method DRL4Route uses an actor-critic architecture, combined with the current behavior learning ability of deep neural networks and the non-differentiable objective optimization ability of reinforcement learning, it can solve the problem of non-differentiable test criteria, but it requires artificially designed reward functions. In the complex delivery personnel spatiotemporal behavior prediction task, the reward function of the reinforcement learning method is difficult to design and difficult to match with the actual scenario. The spatiotemporal behavior prediction accuracy is only improved by about 1%, resulting in the model's spatiotemporal behavior prediction accuracy not being high enough. Summary of the invention

[0004] In order to solve the problem that the above-mentioned reward function is difficult to design and the spatiotemporal behavior prediction accuracy of the model is not high enough, the present invention provides a spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning;

[0005] To achieve the above technical objectives, the details are as follows:

[0006] A spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning, comprising:

[0007] S1. Extracting spatial multi-views from the original spatiotemporal behavior data of the delivery personnel. The spatial multi-views include: spatial area topology map and spatial point topology map;

[0008] The original spatiotemporal behavior data of delivery personnel are obtained from open source datasets;

[0009] Furthermore, the original spatiotemporal behavior data of the delivery personnel are processed, including: the processing of the delivery personnel's own attribute characteristics (attendance rate, vehicle speed, working hours), the processing of the delivery task characteristics (geographic location information, area information, distance information, deadline information), the length limitation of spatiotemporal behavior (i.e., unified data length) and the data set division processing (training set, validation set and test set are divided according to the ratio of 6:2:2);

[0010] In the processed spatiotemporal behavior data of the delivery personnel, both the regions and points are regarded as nodes, from which multiple spatial views are extracted;

[0011] Spatial multi-views include: spatial region topology map G a and spatial point topology G l Among them, G a Includes: the point set of the region, the edge set of the region, the node features of the region, and the edge features of the region; G l Including: point set, point edge set, point node features and point edge features;

[0012] Node characteristics of points The specific definitions are as follows:

[0013]

[0014] In the formula, It is the geographical location information (longitude and latitude) related to the delivery task. and is the ID number and function type (school, commercial area, factory area, and residential area) of the area where the task is located. is the distance between the delivery person and the location of the delivery task, t represents the current moment, and They are the time when the delivery person accepts the delivery task and the deadline of the task. is the final actual completion time of the task;

[0015] Node characteristics of the region The expression is as follows:

[0016]

[0017] In the formula, is the central geographic location information (longitude and latitude) of the region, and is the ID number and functional type of the area (school, commercial area, factory area and residential area), is the distance between the delivery person and the location of the delivery task, t represents the current moment, is the earliest deadline for all delivery tasks in the area;

[0018] Edge features of points The specific definitions are as follows:

[0019] (i≠j, i, j∈W, W is 20)

[0020] In the formula, represents the Euclidean distance between the corresponding positions of node i and node j, represents the difference between the deadlines of tasks corresponding to nodes i and j, represents the connectivity between the corresponding positions of node i and node j, The definition is as follows:

[0021]

[0022] Edge features of the region The specific definitions are as follows:

[0023] (I≠J, I, J∈Z, Z is 10)

[0024] In the formula, It represents the Euclidean distance between the center point of region I corresponding to node i and the center point of region J corresponding to node j. represents the difference between the earliest deadlines of tasks in region I and region J, Represents the connectivity between the corresponding center points of region I and region j;

[0025] In addition to the characteristic information of the point and area, the attributes of the delivery person (such as attendance rate) Speed Working hours ),weather or working day Characteristics also affect the delivery personnel’s spatiotemporal behavior;

[0026] The delivery person's own attributes, weather and working day characteristics are collectively referred to as global features x g , global feature x g The definition is as follows:

[0027]

[0028] S2. Input the spatial multi-view into the personalized adversarial reinforcement learning network to obtain the predicted spatiotemporal behavior of the deliveryman, the actual spatiotemporal behavior of the deliveryman, the reward value of the predicted spatiotemporal behavior, and the expected return of the corresponding state of the spatiotemporal behavior of the deliveryman;

[0029] The personalized adversarial reinforcement learning network includes: a generator, a discriminator, and a value estimator;

[0030] The generator includes: a multi-view encoder and a decoder;

[0031] The multi-view encoder uses an improved graph attention network based on the traditional graph attention network, which introduces a mask attention mechanism, regularization and mean processing of multi-head attention, uses the average operation instead of the splicing operation, and finally uses the ReLU activation function for operation;

[0032] The specific steps are as follows:

[0033] S2.1. Input the spatial multi-view into the generator for multi-view encoding; the details are as follows:

[0034] S2.1.1. Preprocess the data input to the generator to obtain discrete features and continuous features;

[0035] For discrete values, an embedding layer is used to project the discrete values ​​to d disc dimensional vector space to obtain discrete features;

[0036] For continuous values, a linear layer is used to project the continuous value to d conti Dimensional vector space obtains continuous features;

[0037] S2.1.2, concatenate the discrete vector and the continuous vector and input them into the encoder;

[0038] Specifically, the global feature x g The discrete global feature vector and the continuous global feature vector are obtained by projection, and are spliced ​​along the dimension direction (column direction) of the node features and edge features of the region, and the node features and edge features of the point respectively;

[0039] Point Discrete Vector and point continuous vector The splicing process is as follows:

[0040]

[0041] Where W v , W e , b v , b e They are node weight parameter, edge weight parameter, node offset parameter, and edge offset parameter; It is the node feature of the point after splicing; It is the edge feature of the point after splicing;

[0042] Node characteristics of the region and edge features The same splicing process was used to obtain;

[0043] The multi-view encoder takes the node features of the spliced ​​region and the node features of the spliced ​​points as input, and introduces the edge features of the spliced ​​region and the edge features of the spliced ​​points to represent the similarity between the nodes of the region and the points, and uses an improved graph attention network to update the features;

[0044] S2.1.3, use the mask-based attention mechanism to calculate the inter-node attention correlation coefficient of the spliced ​​points;

[0045] Attention correlation coefficient The specific expression is as follows:

[0046]

[0047] Where W1, a v , a e They are the first weight parameter, the node embedding weight parameter, the edge embedding weight parameter, σ is the LeakyReLU activation function, ∥ is the concatenation operation, represents the embedding of node i after concatenation at the kth layer, represents the edge embedding of node i and node j after concatenation at the kth layer;

[0048] S2.1.4, use the softmax mechanism to regularize the attention coefficients of all neighbors of the spliced ​​point node to avoid overfitting; at the same time, use the multi-head attention mechanism to stabilize the learning process;

[0049] The specific expression of the regularized attention coefficient is:

[0050]

[0051] Where N i Indicates area a i The neighbor set of node i, j' is the temporal neighbor node or spatial neighbor node of node i, j'∈N i If and only if

[0052] The specific expression of the multi-head attention mechanism stable learning process is as follows:

[0053]

[0054] Where W2 is the second weight parameter, represents the embedding of node i at the k+1th layer after concatenation, and Y is the number of attention heads;

[0055] S2.1.5. In the last layer (kth layer) of the encoder, the concatenation operation is replaced by the average operation, and the ReLU activation function is used, and the expression is as follows:

[0056]

[0057] Where W2 is the second weight parameter, is the encoding representation of node i in the spatial point topology graph;

[0058] At this point, the encoding representation of the spatial point topology map is obtained The same method is used to obtain the encoding representation of the spatial region topology map

[0059] S2.2. Input the code obtained by the encoder into the decoder to complete the spatiotemporal behavior prediction of the delivery personnel based on the spatial area topology map and the spatial point topology map; the details are as follows:

[0060] S2.2.1. For the prediction of the spatiotemporal behavior of the delivery personnel in the spatial region topology map, at each decoding step, the decoder calculates the probability of the nodes in each candidate region and selects the node with the largest probability as the prediction output of the spatiotemporal behavior of the delivery personnel;

[0061] Specifically, the conditional probability calculated by the chain rule is used to describe the entire decoding process of the deliveryman at the current time t. The expression is as follows:

[0062]

[0063] In the formula, is the region encoding representation output by the encoder; It is to transform the spatiotemporal behavior of the delivery personnel in the generated spatial regional topology map Aggregated; u represents the personalized behavior preference of the deliveryman; θ d represents the learnable parameters in the decoder; is the spatiotemporal behavior of the deliveryman at node i in the spatial regional topology; m∈(1,25);

[0064] For any step r during decoding, the goal is to find the most likely neighbor of the previous step, that is, the output of step r-1, while ensuring that there are no repeated output nodes, and then use the most likely neighbor selected in the current decoding step as the input of the next decoding step to obtain a new hidden state

[0065] Specifically, the hidden state at step r-1 The vector concatenation of and u is used as the query, and the neighbors of the node output in step r-1 are used as the key to calculate the probability distribution between all neighbor nodes. The expression is as follows:

[0066]

[0067] Where W6 and W7 are the sixth and seventh weight parameters, respectively, and v t is the regional learnable weight parameter, Represents the encoding representation of the neighboring node output in the r-1 step in the spatial region topology graph, Represents the hidden state of the r-1th step in the spatial region topology graph. If node j has no output in the previous step, then

[0068] The probability of each output node in the rth step is obtained through softmax operation, and the expression is as follows:

[0069]

[0070] Take the maximum value of all probabilities in step r to form the spatiotemporal behavior prediction output of the delivery personnel in the spatial area topology map in step r The expression is as follows:

[0071]

[0072] S2.2.2. In the prediction of the spatiotemporal behavior of the delivery personnel in the spatial point topology map, the transfer pattern of the spatial region is used to obtain the spatiotemporal behavior prediction of the delivery personnel in the spatial point topology map; the details are as follows:

[0073] The encoding representation of node i in the spatial point topology graph is The transfer mode vector p of the spatial region aoi Concatenate as input Then calculate the probability distribution between all its neighbor nodes The details are as follows:

[0074]

[0075] Where W8 and W9 are the eighth and ninth weight parameters, respectively, and v T is the point-wise learnable weight parameter, Represents the encoding representation of neighbor node j of node i in the spatial point topology graph, Represents the hidden state of the r-1th step in the spatial point topology graph. If node j has no output in the previous step, then

[0076] The probability of each output node in the rth step is obtained through softmax operation, and the expression is as follows:

[0077]

[0078] Where exp(·) is an exponential function with the mathematical constant e as base;

[0079] Take the maximum value of all probabilities in step r to obtain the spatiotemporal behavior prediction output of the delivery personnel in the spatial point topology map in step r The expression is as follows:

[0080]

[0081] In the formula, the argmax function means finding the value that makes Take the maximum value of j;

[0082] S2.3, the discriminator takes the predicted spatiotemporal behavior and the real spatiotemporal behavior as input and distinguishes them according to the reward value of the corresponding spatiotemporal behavior;

[0083] The goal of the discriminator is to distinguish between predicted spatiotemporal behavior and actual spatiotemporal behavior, as follows:

[0084] S2.3.1. The model training is simplified by adding the personalized behavior preference u of the deliveryman to the discriminator. The expression of the discriminator adding the personalized behavior preference u of the deliveryman is:

[0085]

[0086] Where s is the state of the current spatiotemporal behavior node; a is the action selection of the current spatiotemporal behavior; f θ,φ (s,a|u) is the reward estimator, which is expressed as follows:

[0087] f θ,φ (s,a|u)=g θ (s,a|u)+γh φ (s+1|u)-h φ (s|u)

[0088] Where γ is the discount factor, s+1 is the state of the next spatiotemporal behavior node given the state s of the current spatiotemporal behavior node and action a, and g θ (s,a|u) is the reward approximator, h φ (s|u) is the reward shaping item; g θ (s,a|u) is a reward approximator; specifically, the characteristics of the delivery person’s spatiotemporal behavior are represented as a unique encoding vector F a, and concatenate it with the vector learned by the convolutional neural network embedding layer, and use a two-layer feedforward neural network to learn the concatenated vector to obtain g θ (s,a|u);

[0089] h φ (s|u) is the reward shaping term of the state s of the current spatiotemporal behavior node, and the input is the characteristics of the spatiotemporal behavior of the deliveryman F = [F s ; F u ], where F s is the characteristic vector of the deliveryman’s spatiotemporal behavior at the current spatiotemporal behavior node state s, F u is the personalized behavior preference vector of the deliveryman, which is represented by a two-layer feedforward neural network to obtain h φ (s|u);h φ (s+1|u) is the reward shaping term of state s+1, and the input is the characteristics of the deliveryman’s spatiotemporal behavior F = [F s+1 ; F u ], where F s+1 is the characteristic vector of the spatiotemporal behavior of the deliveryman in state s+1;

[0090] S2.3.2. Set the simplified reward value as the entropy regularized policy target to obtain the expected return of the delivery person; the expression is as follows:

[0091] R θ,φ (s,a|u)=log(D θ,φ (s,a|u))-log(1-D θ,φ (s,a|u))

[0092] Where log(·) is a logarithmic function with the mathematical constant e as the base, R θ,φ (s, a|u) is the immediate expected return of the state s and action a of the current spatiotemporal behavior node under the personalized behavior preference u of the delivery person;

[0093] S2.3.3. For real-time and spatial behavior data and predicting spatiotemporal behavior data The batch gradient descent algorithm is used to update the model parameters;

[0094] In this example, the delivery person’s spatiotemporal behavior is divided into M (M∈(0, N), N is a natural integer) state-action pairs to form a training set denoted as i∈{1,2,…,N}}, during the training process, in each round p from Randomly extract a batch of real spatiotemporal behavior data p∈(1,100);

[0095] S2.4, output the result V(s|u) through the value estimator;

[0096] The value estimator V(s|u) is used to calculate the cumulative expected return of the current spatiotemporal behavior node state s under the personalized behavior preference u of the deliveryman. The input is the spatiotemporal behavior feature F = [F s ; F u ], the output is the expected return of the delivery person’s spatiotemporal behavior;

[0097] S3, respectively update the generator, discriminator and value estimator to obtain the optimal value;

[0098] The generator update is trained using the proximal policy optimization algorithm, and the expression is as follows:

[0099]

[0100] In the formula, r G (a|s,u)=π G (a|s,u) / π G,old (a|s,u) represents the probability ratio of the new and old strategies, ∈ is the clipping parameter, clip(·) is the clipping function, is the advantage estimation function; for a delivery person's spatiotemporal behavior data containing T moments, the specific expression of the advantage estimation function is as follows:

[0101]

[0102] Among them, δ t =R θ,φ (s t ,a t |u)+γV(s t+1 |u)-V(s t |u)

[0103] Where γ is the discount factor, parameter λ = 0.95, which is used to balance the bias and variance, R θ,φ (s t ,a t |u) is the expected return of the delivery person’s spatiotemporal behavior at time t, V(s t |u) is the estimated value of the value estimator;

[0104] The update expression of the discriminator is:

[0105]

[0106] Where D θ,φ (s,a|u) represents the output of the discriminator. Given the state s, action a, and personalized behavior preference u of the delivery person at the current spatiotemporal behavior node, the discriminator determines the probability that the data is real spatiotemporal behavior data. represents the expected value of real spatiotemporal behavior data, represents the expected value of the predicted spatiotemporal behavior data;

[0107] The value estimator update expression is:

[0108]

[0109] In the formula, Q(s,a|u) represents the reward value of the state s and action a of the current spatiotemporal behavior node based on the personalized behavior preference u of the delivery person, and is used as and The sum of θ v ,θ v,old Represent the new and old parameters of the value estimator respectively;

[0110] Finally, the optimal value is obtained; the details are as follows:

[0111] Based on the updating process of the generator, the discriminator and the value estimator, the three models compete with each other when training the generator, the discriminator and the value estimator, so as to continuously improve the three models until they reach the best state; the three models use the training set to optimize and update the model parameters, use the validation set to select the parameters, and use the test set to evaluate the generalization performance of the model;

[0112] The goal of the generator update process is to find the spatiotemporal behavior policy that produces the maximum reward;

[0113] The goal of the discriminator update process is to give a reward function that can distinguish between real spatiotemporal behavior and predicted spatiotemporal behavior based on the reward value;

[0114] The goal of the value estimator update process is to provide the generator with value information about the current spatiotemporal state, helping the generator to more effectively optimize the spatiotemporal behavior strategy;

[0115] Beneficial effects of the present invention

[0116] 1. By introducing spatial multi-views and personalized behavioral preferences of delivery personnel into the adversarial reinforcement learning network, the present invention can restore the reward function from the historical spatiotemporal behavior data of the delivery personnel, effectively avoid the problem of difficult design of reward functions in reinforcement learning, and learn optimized reward functions and strategy functions at the same time, thereby improving the interpretability of the model and the accuracy of prediction.

[0117] 2. The present invention introduces spatial multi-views, including a spatial area topology map and a spatial point topology map. The transfer pattern of the spatial area is obtained by predicting the spatiotemporal behavior of the delivery personnel in the spatial area topology map. The transfer pattern of the spatial area is used to guide the spatiotemporal behavior prediction process of the delivery personnel in the spatial point topology map, thereby improving the prediction accuracy.

[0118] 3. In the process of predicting the spatiotemporal behavior of delivery personnel, the present invention takes into account the selection preferences of delivery personnel between different locations, introduces the personalized behavior preferences of delivery personnel, reduces the large search space in spatiotemporal behavior prediction, speeds up calculation efficiency, and improves the accuracy of spatiotemporal behavior prediction of delivery personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0120] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0121] This example uses the public LaDe logistics distribution dataset, which is the first comprehensive last-mile delivery dataset collected by Cainiao Network (one of the largest logistics companies in China). It has three notable features:

[0122] The scale is huge, involving 10,677,000 packages delivered by 21,000 couriers in 6 months of actual operation;

[0123] Comprehensive information: The dataset provides raw package information, such as the location and time requirements of the package, as well as task event information, recording the time and location of the delivery person when events such as task acceptance and task completion occur;

[0124] Diversity,The dataset contains data from various scenarios such as package pickup and delivery, as well as data from multiple cities in China (Shanghai, Chongqing, Hangzhou), each of which has unique spatiotemporal patterns due to its unique characteristics such as population, economy, and geographical location.

[0125] The real city express delivery collection dataset of Shanghai is selected from the LaDe logistics dataset. The Shanghai dataset contains the collection records of 4502 delivery personnel in Shanghai, China for six months from May 2023 to October 2023. In the dataset, the training set, validation set, and test set are divided in a 6:2:2 ratio in chronological order. The detailed information about the dataset is shown in Table 1 below.

[0126] Table 1 Dataset details

[0127] City time Number of people Number of spatiotemporal behaviors Average arrival time Average number of pickups Shanghai 6 months 4502 people 96,000 151 minutes 15 items

[0128] See also Figure 1, a spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning, comprising:

[0129] S1. Extracting spatial multi-views from the original spatiotemporal behavior data of the delivery personnel. The spatial multi-views include: spatial area topology map and spatial point topology map;

[0130] Specifically, based on the LaDe logistics delivery dataset, the spatiotemporal behavior data of each deliveryman in Shanghai, China from May to October 2023 are obtained, including: spatiotemporal constraints of uncollected tasks (including: package location and promised collection time), characteristics of the most recent collection tasks (including: collection time and collection location), and characteristics of deliverymen (including: total working days and average number of collections per day);

[0131] Furthermore, the original spatiotemporal behavior data of the delivery personnel are processed, including: the processing of the delivery personnel's own attribute characteristics (attendance rate, vehicle speed, working hours), the processing of the delivery task characteristics (geographic location information, area information, distance information, deadline information), the length limitation of spatiotemporal behavior (i.e., unified data length) and the data set division processing (training set, validation set and test set are divided according to the ratio of 6:2:2);

[0132] In the processed spatiotemporal behavior data of the delivery personnel, both the regions and points are regarded as nodes, from which multiple spatial views are extracted;

[0133] Spatial multi-views include: spatial region topology map G a and spatial point topology G l Among them, G a Includes: the point set of the region, the edge set of the region, the node features of the region, and the edge features of the region; G l Including: point set, point edge set, point node features and point edge features;

[0134] Node characteristics of points The specific definitions are as follows:

[0135]

[0136] In the formula, It is the geographical location information (longitude and latitude) related to the delivery task. and is the ID number and function type (school, commercial area, factory area, and residential area) of the area where the task is located. is the distance between the delivery person and the location of the delivery task, t represents the current moment, and They are the time when the delivery person accepts the delivery task and the deadline of the task. is the final actual completion time of the task;

[0137] Node characteristics of the region The expression is as follows:

[0138]

[0139] In the formula, is the central geographic location information (longitude and latitude) of the region, and is the ID number and functional type of the area (school, commercial area, factory area and residential area), is the distance between the delivery person and the location of the delivery task, t represents the current moment, is the earliest deadline for all delivery tasks in the area;

[0140] Edge features of points The specific definitions are as follows:

[0141] (i≠j, i, j∈W, W is 20)

[0142] In the formula, represents the Euclidean distance between the corresponding positions of node i and node j, represents the difference between the deadlines of tasks corresponding to nodes i and j, represents the connectivity between the corresponding positions of node i and node j, The definition is as follows:

[0143]

[0144] Edge features of the region The specific definitions are as follows:

[0145] (I≠J, I, J∈Z, Z is 10)

[0146] In the formula, It represents the Euclidean distance between the center point of region I corresponding to node i and the center point of region J corresponding to node j. represents the difference between the earliest deadlines of tasks in region I and region J, Represents the connectivity between the corresponding center points of region I and region J;

[0147] In addition to the characteristic information of the point and area, the attributes of the delivery person (such as attendance rate) Speed Working hours ),weather or working day Characteristics also affect the delivery personnel’s spatiotemporal behavior;

[0148] The delivery person's own attributes, weather and working day characteristics are collectively referred to as global features x g , global feature x g The definition is as follows:

[0149]

[0150] S2. Input the spatial multi-view into the personalized adversarial reinforcement learning network to obtain the predicted spatiotemporal behavior of the deliveryman, the actual spatiotemporal behavior of the deliveryman, the reward value of the predicted spatiotemporal behavior, and the expected return of the corresponding state of the spatiotemporal behavior of the deliveryman;

[0151] The personalized adversarial reinforcement learning network includes: a generator, a discriminator, and a value estimator;

[0152] The generator includes: a multi-view encoder and a decoder;

[0153] The multi-view encoder uses an improved graph attention network based on the traditional graph attention network, which introduces a mask attention mechanism, regularization and mean processing of multi-head attention, uses the average operation instead of the splicing operation, and finally uses the ReLU activation function for operation;

[0154] The specific steps are as follows:

[0155] S2.1. Input the spatial multi-view into the generator for multi-view encoding, as follows:

[0156] S2.1.1. Preprocess the data input to the generator to obtain discrete features and continuous features;

[0157] For discrete values, an embedding layer is used to project the discrete values ​​to d disc dimensional vector space to obtain discrete features;

[0158] For continuous values, a linear layer is used to project the continuous value to d conti Dimensional vector space obtains continuous features;

[0159] In this embodiment, the discrete values ​​include: the ID number of the region, the function type, the attendance rate of the deliveryman, the speed of the deliveryman, the working hours of the deliveryman, the weather characteristics, and the working day characteristics. The discrete values ​​are projected onto the d disc dimensional vector space to obtain discrete features;

[0160] The continuous values ​​include: longitude and latitude information, distance information between the delivery person and the delivery task location, current time information, time when the delivery person accepts the delivery task, deadline of the delivery task, and final actual completion time of the delivery task. The continuous values ​​are projected onto d using a linear layer. conti Dimensional vector space obtains continuous features;

[0161] S2.1.2, concatenate the discrete vector and the continuous vector and input them into the encoder;

[0162] The global feature x g The discrete global feature vector and the continuous global feature vector are obtained by projection, and are spliced ​​along the dimension direction (column direction) of the node features and edge features of the region, and the node features and edge features of the point respectively;

[0163] Point Discrete Vector and point continuous vector The splicing process is as follows:

[0164]

[0165] Where W v , W e , b v , b e They are node weight parameter, edge weight parameter, node offset parameter, and edge offset parameter; It is the node feature of the point after splicing; is the edge feature of the spliced ​​point; the model is trained through the back propagation algorithm, W v , W e , b v , b e Optimized during training;

[0166] Node characteristics of the region and edge features The same splicing process is used to obtain the nodes. The multi-view encoder takes the node features of the spliced ​​region and the node features of the spliced ​​points as input, and introduces the edge features of the spliced ​​region and the edge features of the spliced ​​points to represent the similarity between the nodes of the region and the points, and uses an improved graph attention network to update the features.

[0167] S2.1.3, use the mask-based attention mechanism to calculate the inter-node attention correlation coefficient of the spliced ​​points;

[0168] The key to the calculation process is to calculate the attention correlation coefficient between nodes. The specific expression is as follows:

[0169]

[0170] Where W1, a v , a e They are the first weight parameter, the node embedding weight parameter, the edge embedding weight parameter, σ is the LeakyReLU activation function, ∥ is the concatenation operation, represents the embedding of node i after concatenation at the kth layer, represents the edge embedding of node i and node j after concatenation at the kth layer; the model is trained by the back propagation algorithm, W1, a v , a e Optimized during training;

[0171] S2.1.4. Use the softmax mechanism to regularize the attention coefficients of all neighbors of the spliced ​​point node to avoid overfitting; at the same time, use the multi-head attention mechanism to stabilize the learning process; the specific expression of the regularized attention coefficient is as follows:

[0172]

[0173] Where N i Indicates area a i The neighbor set of node i, j' is the temporal neighbor node or spatial neighbor node of node i, j'∈N i If and only if

[0174] In order to stabilize the learning process of the attention mechanism, a multi-head attention mechanism is introduced, using Y attention heads and a regularized attention coefficient Update, in this embodiment, 8 attention heads are taken, and the specific expression is as follows:

[0175]

[0176] Where W2 is the second weight parameter, represents the embedding of node i at the point after concatenation at the k+1th layer; the model is trained using the back propagation algorithm and is optimized during the W2 training process;

[0177] The edge representation of the spliced ​​points is updated through linear transformation. The specific expression is as follows:

[0178]

[0179] Wherein, W3, W4, and W5 are the third weight parameter, the fourth weight parameter, and the fifth weight parameter, respectively. represents the edge embedding of the spliced ​​node i and the spliced ​​node j at the k+1th layer; represents the embedding of node j at the kth layer of concatenation; represents the edge embedding of the spliced ​​node i and the spliced ​​node j at the kth layer point; the model is trained by the back propagation algorithm, and W3, W4, and W5 are optimized during the training process; in this embodiment, k is 3;

[0180] S2.1.5. In the last layer of the encoder, the concatenation operation is replaced by the average operation, and the ReLU activation function is used, and the expression is as follows:

[0181]

[0182] Where W2 is the second weight parameter, is the encoding representation of node i in the spatial point topology graph;

[0183] At this point, the encoding representation of the spatial point topology map is obtained The same method is used to obtain the encoding representation of the spatial region topology map

[0184] S2.2, input the code obtained by the encoder into the decoder to complete the spatiotemporal behavior prediction of the delivery personnel based on the spatial area topology map and the spatial point topology map;

[0185] The codes obtained by the encoder include: codes of the spatial point topology map and codes of the spatial area topology map;

[0186] Among them, the spatiotemporal behavior prediction results of the delivery personnel in the spatial area topology map can be used as important guiding information for the spatiotemporal behavior prediction of the delivery personnel in the spatial point topology map; based on the long short-term memory network, a masked attention mechanism is designed to simulate the spatiotemporal behavior prediction process of the delivery personnel, as follows:

[0187] S2.2.1. For the prediction of the spatiotemporal behavior of the delivery personnel in the spatial region topology map, at each decoding step, the decoder calculates the probability of the nodes in each candidate region and selects the node with the largest probability as the prediction output of the spatiotemporal behavior of the delivery personnel, as follows:

[0188] The conditional probability calculated by the chain rule is used to describe the entire decoding process of the deliveryman at the current time t. The specific expression is as follows:

[0189]

[0190] In the formula, is the region encoding representation output by the encoder; It is to transform the spatiotemporal behavior of the delivery personnel in the generated spatial regional topology map Aggregated; u represents the personalized behavior preference of the deliveryman; θ d represents the learnable parameters in the decoder; is the spatiotemporal behavior of the deliveryman at node i in the spatial regional topology; m∈(1,25);

[0191] For any step r, r∈m during decoding, the goal is to find the most likely neighbor of the previous step, that is, the output of step r-1, while ensuring that there are no repeated output nodes, and then use the most likely neighbor selected in the current decoding step as the input of the next decoding step to obtain a new hidden state Specifically, the hidden state at step r-1 The concatenation of and u is used as the query, and the neighbors of the node output in the previous step are used as the key to calculate the probability distribution between all neighbor nodes. The details are as follows:

[0192]

[0193] Where W6 and W7 are the sixth and seventh weight parameters, respectively, and v t is the regional learnable weight parameter, Represents the encoding representation of the neighboring node output in the r-1 step in the spatial region topology graph, represents the hidden state of the r-1th step in the spatial region topology graph. If node j has no output in the previous step, it is represented as The model is trained using the back propagation algorithm, W6, W7, v t Optimized during training;

[0194] The probability of each output node in the rth step is obtained through softmax operation, and the expression is as follows:

[0195]

[0196] Take the maximum value of all probabilities in step r to form the spatiotemporal behavior prediction output of the deliveryman in the spatial area topology graph in step r. The expression is as follows:

[0197]

[0198] S2.2.2. In the prediction of the spatiotemporal behavior of the delivery personnel in the spatial point topology map, the transfer pattern of the spatial region is used to obtain the spatiotemporal behavior prediction of the delivery personnel in the spatial point topology map; the details are as follows:

[0199] Prediction of the spatiotemporal behavior of the delivery personnel in the spatial point topology map. In the rth step of decoding, the transfer pattern of the spatial region is added to calculate the conditional probability of each point. Specifically, the encoding representation of node i in the spatial point topology map is The transfer mode vector p of the spatial region aoi Concatenate as input Then calculate the probability distribution between all its neighbor nodes The details are as follows:

[0200]

[0201] Where W8 and W9 are the eighth and ninth weight parameters, respectively, and v T is the point-wise learnable weight parameter, Represents the encoding representation of neighbor node j of node i in the spatial point topology graph, Represents the hidden state of the r-1th step in the spatial point topology graph. If node j has no output in the previous step, it is represented as The model is trained using the back propagation algorithm. T Optimized during training;

[0202] The probability of each output node in the rth step is obtained through softmax operation, and the expression is as follows:

[0203]

[0204] Where exp(·) is an exponential function with the mathematical constant e as base;

[0205] Take the maximum value of all probabilities in step r to obtain the spatiotemporal behavior prediction output of the delivery personnel in the spatial point topology map in step r The expression is as follows:

[0206]

[0207] In the formula, the argmax function means finding the value that makes Take the maximum value of j;

[0208] S2.3, the discriminator takes the predicted spatiotemporal behavior and the real spatiotemporal behavior as input and distinguishes them according to the reward value of the corresponding spatiotemporal behavior;

[0209] The goal of the discriminator is to distinguish between predicted spatiotemporal behavior and actual spatiotemporal behavior, as follows:

[0210] S2.3.1. The model training is simplified by adding the personalized behavior preference u of the deliveryman to the discriminator. The expression of the discriminator adding the personalized behavior preference u of the deliveryman is:

[0211]

[0212] Where s is the state of the current spatiotemporal behavior node; a is the action selection of the current spatiotemporal behavior; f θ,φ (s,a|u) is the reward estimator, which is expressed as follows:

[0213] f θ,φ (s,a|u)=g θ (s,a|u)+γh φ (s+1|u)-h φ (s|u)

[0214] Where γ is the discount factor, γ∈[0,1], s+1 is the state s of the current spatiotemporal behavior node and the state of the next spatiotemporal behavior node of action a, and g θ(s,a|u) is the reward approximator, h φ (s|u) is the reward shaping item; g θ (s,a|u) is a reward approximator; specifically, the characteristics of the delivery person’s spatiotemporal behavior are represented as a unique encoding vector F a , and concatenate it with the vector learned by the convolutional neural network embedding layer, and use a two-layer feedforward neural network to learn the concatenated vector to obtain g θ (s,a|u);

[0215] h φ (s|u) is the reward shaping term of the state s of the current spatiotemporal behavior node, and the input is the characteristics of the spatiotemporal behavior of the deliveryman F = [F s ; F u ], where F s is the characteristic vector of the deliveryman’s spatiotemporal behavior at the current spatiotemporal behavior node state s, F u is the personalized behavior preference vector of the deliveryman, which is represented by a two-layer feedforward neural network to obtain h φ (s|u);h φ (s+1|u) is the reward shaping term of state s+1, and the input is the characteristics of the deliveryman’s spatiotemporal behavior F = [F s+1 ; F u ], where F s+1 is the characteristic vector of the spatiotemporal behavior of the deliveryman in state s+1;

[0216] S2.3.2. Set the differentiated reward value as the entropy regularized strategy target to obtain the expected return of the delivery person; the expression is as follows:

[0217] R θ,φ (s,a|u)=log(D θ,φ (s,a|u))-log(1-D θ,φ (s,a|u))

[0218] Where log(·) is a logarithmic function with the mathematical constant e as the base, R θ,φ (s, a|u) is the immediate expected return of the state s and action a of the current spatiotemporal behavior node under the personalized behavior preference u of the delivery person;

[0219] S2.3.3. For real-time and spatial behavior data and predicting spatiotemporal behavior data The batch gradient descent algorithm is used to update the model parameters;

[0220] In this example, the delivery person’s spatiotemporal behavior is divided into M (M∈(0, N), N is a natural integer) state-action pairs to form a training set denoted as i∈{1,2,…,N}}, during the training process, in each round p from Randomly extract a batch of real spatiotemporal behavior data p∈(1,100);

[0221] S2.4, output the result V(s|u) through the value estimator;

[0222] The value estimator V(s|u) is used to calculate the cumulative expected return of the current spatiotemporal behavior node state s under the personalized behavior preference u of the deliveryman. The input is the spatiotemporal behavior feature F = [F s ; F u ], the output is the expected return of the delivery person’s spatiotemporal behavior;

[0223] S3, respectively update the generator, discriminator and value estimator to obtain the optimal value;

[0224] The generator update is trained using the proximal policy optimization algorithm, and the expression is as follows:

[0225]

[0226] In the formula, r G (a|s,u)=π G (a|s,u) / π G,old (a|s,u) represents the probability ratio of the new and old strategies, ∈ is the clipping parameter, ∈∈(0.1, 0.3), clip(·) is the clipping function, is the advantage estimation function; for a delivery person's spatiotemporal behavior data containing T moments, T∈[0,120]; the specific expression of the advantage estimation function is as follows:

[0227]

[0228] Among them, δ t =R θ,φ (s t ,a t |u)+γV(s t+1 |u)-V(s t |u)

[0229] Where γ is the discount factor, parameter λ = 0.95, which is used to balance the bias and variance, R θ,φ (s t ,a t |u) is the expected return of the delivery person’s spatiotemporal behavior at time t, V(s t |u) is the estimated value of the value estimator; T i ∈T;

[0230] The update expression of the discriminator is:

[0231]

[0232] Where D θ,φ (s,a|u) represents the output of the discriminator. Given the state s, action a, and personalized behavior preference u of the delivery person at the current spatiotemporal behavior node, the discriminator determines the probability that the data is real spatiotemporal behavior data. represents the expected value of real spatiotemporal behavior data, represents the expected value of the predicted spatiotemporal behavior data;

[0233] The value estimator update expression is:

[0234]

[0235] In the formula, Q(s,a|u) represents the reward value of the state s and action a of the current spatiotemporal behavior node based on the personalized behavior preference u of the delivery person, and is used as and The sum of θ v ,θ v,old Represent the new and old parameters of the value estimator respectively;

[0236] Finally, the optimal value is obtained; the details are as follows:

[0237] Based on the updating process of the generator, the discriminator and the value estimator, the three models compete with each other when training the generator, the discriminator and the value estimator, so as to continuously improve the three models until they reach the best state; the three models use the training set to optimize and update the model parameters, use the validation set to select the parameters, and use the test set to evaluate the generalization performance of the model;

[0238] The goal of the generator update process is to find the spatiotemporal behavior policy that produces the maximum reward;

[0239] The goal of the discriminator update process is to give a reward function that can distinguish between real spatiotemporal behavior and predicted spatiotemporal behavior based on the reward value;

[0240] The goal of the value estimator update process is to provide the generator with value information about the current spatiotemporal state, helping the generator to more effectively optimize the spatiotemporal behavior strategy;

[0241] The comparison of the test results of the deliveryman spatiotemporal behavior prediction method based on spatial multi-view personalized adversarial reinforcement learning proposed in the present invention and the relevant mainstream methods in the field of deliveryman spatiotemporal behavior prediction on the same Shanghai dataset is shown in Table 1:

[0242] Table 1 Effect comparison results

[0243]

[0244]

[0245] In the table, HR@1 is the hit rate, ACC@3 is the accuracy, KRC is the Kendall correlation coefficient, LSD is the position square deviation, and ED is the edit distance;

[0246] Among them, the higher the HR@1, ACC@3, and KRC, and the lower the LSD and ED, the better the spatiotemporal behavior prediction performance of the method. It can be seen that this method is superior to the current best method in all indicators, and the indicator effect is improved by 1.6% to 3.6%, which proves the effectiveness of this method.

[0247] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as admitting or implying in any form that the information constitutes prior art known to those skilled in the art. The changes and modifications made by the technical personnel in this field do not deviate from the spirit and scope of the present invention, and should all be within the protection scope of the claims attached to the present invention.

Claims

1. A spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning, characterized in that: The following steps are involved: S1, extracting spatial multi-views from the original delivery personnel spatiotemporal behavior data; The spatial multi-views include: a spatial area topology map and a spatial point topology map; S2. Input the spatial multi-view into the personalized adversarial reinforcement learning network to obtain the predicted spatiotemporal behavior of the deliveryman, the actual spatiotemporal behavior of the deliveryman, the reward value of the predicted spatiotemporal behavior, and the expected return of the corresponding state of the spatiotemporal behavior of the deliveryman; The personalized adversarial reinforcement learning network includes: a generator, a discriminator and a value estimator; The generator includes: a multi-view encoder and a decoder; The multi-view encoder is based on the traditional graph attention network as the basic network structure, introduces the mask attention mechanism, multi-head attention regularization and mean processing; and uses the average operation instead of the splicing operation; and finally uses the ReLU activation function; S3. Update the generator, discriminator and value estimator respectively and finally get the optimal value.

2. The spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 1, characterized in that: The spatial region topology map includes: a point set of the region, an edge set of the region, a node feature of the region, and an edge feature of the region; the spatial point topology map includes: a point set, an edge set of the point, a node feature of the point, and an edge feature of the point; The expression of the node characteristics of the region is as follows: In the formula, It is the central geographical location information of the region. and is the region's ID number and feature type, is the distance between the delivery person and the location of the delivery task, t represents the current moment, is the earliest deadline for all delivery tasks in the area; The edge features of the region are specifically defined as follows: (I≠J, I, J∈Z, Z is 10) In the formula, It represents the Euclidean distance between the center point of region I corresponding to node i and the center point of region J corresponding to node j. represents the difference between the earliest deadlines of tasks in region I and region J, Represents the connectivity between the corresponding center points of region I and region J; The node features of the points are specifically defined as follows: In the formula, It is the geographic location information related to the delivery task. and is the ID number and function type of the area where the task is located. is the distance between the delivery person and the location of the delivery task, t represents the current moment, and They are the time when the delivery person accepts the delivery task and the deadline of the task. is the final actual completion time of the task; The edge features of the points are specifically defined as follows: (i≠j, i, j∈W, W is 20) In the formula, represents the Euclidean distance between the corresponding positions of node i and node j, represents the difference between the deadlines of tasks corresponding to nodes i and j, represents the connectivity between the corresponding positions of node i and node j, The definition is as follows:

3. The spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 1, characterized in that: The specific steps of inputting the spatial multi-view into the personalized adversarial reinforcement learning network to obtain the predicted spatiotemporal behavior of the deliveryman, the actual spatiotemporal behavior of the deliveryman, the reward value of the predicted spatiotemporal behavior, and the expected return of the corresponding state of the spatiotemporal behavior of the deliveryman are as follows: S2.1, inputting spatial multi-views into the generator for multi-view encoding; S2.2, input the multi-view code into the decoder to complete the spatiotemporal behavior prediction of the delivery personnel based on the spatial area topology map and the spatial point topology map; S2.3, the discriminator takes the predicted spatiotemporal behavior and the real spatiotemporal behavior as input and distinguishes them according to the reward value of the corresponding spatiotemporal behavior; S2.

4. Output the result through the value estimator.

4. The spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 1, characterized in that: The generator, discriminator and value estimator are updated as follows: The generator update is trained using the proximal policy optimization algorithm, and the expression is as follows: In the formula, r G (a|s,u)=π G (a|s,u) / π G,old (a|s,u) represents the probability ratio of the new and old strategies; ∈ is the clipping parameter; clip(·) is the clipping function; is the advantage estimation function, expressed as follows: where, δ t = R θ,φ (s t , a t |u) + γV(s t+1 |u) - V(s t |u) Where γ is the discount factor, parameter λ = 0.95, which is used to balance the bias and variance, R θ,φ (s t ,a t |u) is the expected return of the delivery person’s spatiotemporal behavior at time t, V(s t |u) is the estimated value of the value estimator; The update expression of the discriminator is: Where D θ,φ (s,a|u) represents the output of the discriminator. Given the state s, action a, and personalized behavior preference u of the delivery person at the current spatiotemporal behavior node, the discriminator determines the probability that the data is real spatiotemporal behavior data. represents the expected value of real spatiotemporal behavior data, represents the expected value of the predicted spatiotemporal behavior data; The value estimator update expression is: In the formula, Q(s,a|u) represents the reward value of the state s and action a of the current spatiotemporal behavior node based on the personalized behavior preference u of the delivery person, and is used as and The sum of θ v ,θ v,old Represent the new and old parameters of the value estimator respectively.

5. The spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 3, characterized in that: The steps of inputting spatial multi-views into the generator for multi-view encoding are as follows: S2.1.

1. Preprocess the data input to the generator to obtain discrete features and continuous features; For discrete values, an embedding layer is used to project the discrete values ​​to d disc dimensional vector space to obtain discrete features; For continuous values, a linear layer is used to project the continuous value to d conti Dimensional vector space obtains continuous features; S2.1.2, concatenate the discrete vector and the continuous vector and input them into the encoder; S2.1.3, use the mask-based attention mechanism to calculate the inter-node attention correlation coefficient of the spliced ​​points; Attention correlation coefficient The specific expression is as follows: Where, W1, a v , a e They are the first weight parameter, the node embedding weight parameter, the edge embedding weight parameter, σ is the LeakyReLU activation function, ∥ is the concatenation operation, represents the embedding of node i after concatenation at the kth layer, represents the edge embedding of node i and node j after concatenation at the kth layer; S2.1.4, use the softmax mechanism to regularize the attention coefficients of all neighbors of the spliced ​​point node to avoid overfitting; at the same time, use the multi-head attention mechanism to stabilize the learning process; S2.1.

5. In the last layer of the encoder, replace the concatenation operation with the average operation and use the ReLU activation function.

6. The spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 3, characterized in that: The steps of inputting the multi-view code into the decoder to complete the spatiotemporal behavior prediction of the delivery personnel based on the spatial area topology map and the spatial point topology map are as follows: S2.2.

1. For the prediction of the spatiotemporal behavior of the deliveryman in the spatial region topology map, at each decoding step, the decoder calculates the probability of the nodes in each candidate region and selects the node with the largest probability as the prediction output of the spatiotemporal behavior of the deliveryman; the expression is as follows: In the formula, The probability of each output node in the rth step of the spatial region topology graph is obtained by softmax operation; the argmax function means that among all possible values ​​of node j, find Take the maximum value of j; S2.2.

2. In the prediction of the spatiotemporal behavior of the delivery personnel in the spatial point topology map, the transfer pattern of the spatial region is used to obtain the spatiotemporal behavior prediction of the delivery personnel in the spatial point topology map; the expression is as follows: In the formula, The probability of each output node in the rth step of the spatial point topology graph is obtained by softmax operation; the argmax function means that among all possible values ​​of node j, find Take the j with the largest value.

7. The spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 3, characterized in that: The discriminator takes the predicted spatiotemporal behavior and the real spatiotemporal behavior as input, and distinguishes them according to the reward value of the corresponding spatiotemporal behavior as follows: S2.3.

1. Simplify model training by adding the personalized behavior preference u of the delivery person to the discriminator; The expression of the discriminator that adds the personalized behavior preference u of the delivery person is: Where s is the state of the current spatiotemporal behavior node; a is the action choice of the current spatiotemporal behavior; f θ,φ (s,a|u) is the reward estimator, which is expressed as follows: f θ,φ (s,a|u)=g θ (s,a|u)+γh φ (s+1|u)-h φ (s|u) Where γ is the discount factor, s+1 is the state of the next spatiotemporal behavior node given the state s of the current spatiotemporal behavior node and action a, and g θ (s,a|u) is the reward approximator, h φ (s|u) is the reward shaping item; S2.3.

2. Set the simplified reward value as the entropy regularized policy target to obtain the expected return of the delivery person; the expression is as follows: R θ,φ (s,a|u)=log(D θ,φ (s,a|u))-log(1-D θ,φ (s,a|u)) Where log(·) is a logarithmic function with the mathematical constant e as the base, R θ,φ (s, a|u) is the immediate expected return of the state s and action a of the current spatiotemporal behavior node under the personalized behavior preference u of the delivery person; S2.3.

3. For real-time and spatial behavior data and predicting spatiotemporal behavior data The batch gradient descent algorithm is used to update the model parameters.

Images