A spatio-temporal behavior prediction method based on multi-view personalized inverse reinforcement learning

Abstract

The application discloses a kind of based on multi-view personalized inverse reinforcement learning's space-time behavior prediction method, it is related to space-time prediction field.The present application includes: extracting spatial multi-view from original deliveryman space-time behavior data;Space multi-view is input into personalized confrontation inverse reinforcement learning network, including generator, discriminator and value estimator, generator includes multi-view encoder and decoder, input is obtained area and point position representation after multi-view encoder, and decoder outputs deliveryman space-time behavior;Discriminator carries out training with prediction and real space-time behavior as input;Value estimator calculates the expected return of current state;Generator, discriminator and value estimator are alternately trained.The present application introduces multi-view and personalized behavior preference into confrontation inverse reinforcement learning network, learns reward function from historical behavior data and predicts the space-time behavior of deliveryman, enhances the explainability of model and improves the accuracy of space-time behavior prediction.

Description

Technical Field

[0001] This invention relates to the field of path planning, and more specifically to a spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning. Background Technology

[0002] In recent years, on-demand delivery services (such as express delivery and food delivery) have flourished, becoming an important link between various stages of production and consumption. In delivery service scenarios, accurately predicting the spatiotemporal behavior of delivery personnel is a key aspect of achieving intelligent delivery. The problem of predicting the spatiotemporal behavior of delivery personnel involves highly complex spatiotemporal factors, fundamentally different from traditional combinatorial optimization problems that seek to minimize predefined time or distance costs, and prediction problems that involve selecting the next location from a location vocabulary. Most existing spatiotemporal behavior prediction technologies can only model simple spatiotemporal features, ignoring the influence of delivery personnel's personalized behavioral preferences. This results in a significant deviation between the actual and predicted spatiotemporal behavior of delivery personnel, making it difficult to meet the required accuracy for intelligent delivery applications.

[0003] Many existing techniques for predicting the spatiotemporal behavior of delivery personnel, such as DeepRoute (which utilizes a Transformer-like encoder and an attention-based recurrent decoder), RankETPA (which uses a recurrent neural network for prediction), and Graph2Route (which proposes prediction based on a dynamic spatiotemporal graph), suffer from a drawback: the test criterion is nondifferentiable. The alternative training criterion used during model training differs from the test criterion, leading to poor model performance. While the reinforcement learning method DRL4Route uses an actor-critic architecture, combining the behavioral learning capabilities of current deep neural networks with the nondifferentiable objective optimization capabilities of reinforcement learning to address the nondifferentiability issue, it requires manually designed reward functions. In complex spatiotemporal behavior prediction tasks, the reward functions of reinforcement learning methods are difficult to design and hard to match with real-world scenarios, resulting in only about a 1% improvement in spatiotemporal behavior prediction accuracy, thus failing to achieve high overall model accuracy. Summary of the Invention

[0004] To address the challenges of designing reward functions and the insufficient accuracy of spatiotemporal behavior prediction in the aforementioned models, this invention provides a spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning.

[0005] To achieve the above technical objectives, the specific measures are as follows:

[0006] A spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning includes:

[0007] S1. Extract spatial multi-views from the original deliveryman spatiotemporal behavior data. The spatial multi-views include: spatial region topology map and spatial point topology map.

[0008] The original spatiotemporal behavior data of delivery personnel was obtained from an open-source dataset;

[0009] Furthermore, the original spatiotemporal behavior data of delivery personnel is processed, including: processing of delivery personnel's own attribute characteristics (attendance rate, vehicle speed, working hours), processing of delivery task characteristics (geographical location information, location information, distance information, deadline information), limiting the length of spatiotemporal behavior (i.e., unifying the data length), and processing of dataset partitioning (the training set, validation set, and test set are divided in a 6:2:2 ratio).

[0010] In the processed spatiotemporal behavior data of delivery personnel, both regions and locations are treated as nodes, from which spatial multi-views are extracted;

[0011] Spatial multi-view includes: spatial region topology map Spatial point topology map ;in, This includes: the set of points in the region, the set of edges in the region, the node features of the region, and the edge features of the region; This includes: the set of points, the set of edges of points, the node features of points, and the edge features of points;

[0012] Node characteristics of the point The specific definitions are as follows:

[0013]

[0014] In the formula, It is the geographic location information (longitude and latitude) related to the delivery task. and The area where the mission is located Number and function type (school, commercial area, factory area and residential area). It is the distance between the delivery person and the location of the delivery task. Representing the current moment, and These are the time when the delivery person accepts the delivery task and the deadline for the task. It is the final, actual completion time of the task;

[0015] Node characteristics of the region The expression is as follows:

[0016]

[0017] In the formula, It is the central geographical location information (longitude and latitude) of the region. and It is a region Number and function type (school, commercial area, factory area and residential area). It is the distance between the delivery person and the location of the delivery task. Representing the current moment, This is the earliest deadline for all delivery tasks in the region;

[0018] Edge features of the point The specific definitions are as follows:

[0019] (i≠j, i, j∈W)

[0020] In the formula, Represents a node and nodes Euclidean distance at corresponding positions Represents a node and nodes The difference between the corresponding task deadlines, Represents a node and nodes The connectivity between corresponding positions, the The definition is as follows:

[0021]

[0022] Edge features of the region The specific definitions are as follows:

[0023] (I≠J, I, J∈Z)

[0024] In the formula, Indicates the region corresponding to node i and the region corresponding to node j Euclidean distance corresponding to the center point position, Indicates the area and region The difference between the earliest deadlines of the tasks. Indicates the area and region Connectivity between corresponding center point locations;

[0025] In addition to location and area characteristics, delivery personnel's own attributes (such as attendance rate) Speed Working hours ),weather or weekdays Characteristics also influence the delivery person's spatiotemporal behavior;

[0026] The delivery person's own attributes, weather conditions, and workday characteristics are collectively referred to as global characteristics. global features The definition is as follows:

[0027] ;

[0028] S2. Input the spatial multi-view into the personalized adversarial inverse reinforcement learning network to obtain the deliveryman's predicted spatiotemporal behavior, the deliveryman's actual spatiotemporal behavior, the reward value of the predicted spatiotemporal behavior, and the expected reward of the deliveryman's spatiotemporal behavior corresponding to the state.

[0029] Personalized adversarial inverse reinforcement learning networks include: a generator, a discriminator, and a value estimator;

[0030] The generator includes: a multi-view encoder and a decoder;

[0031] The multi-view encoder adopts an improved graph attention network based on the traditional graph attention network, which introduces mask attention mechanism, regularization and mean processing of multi-head attention, uses averaging operation to replace concatenation operation, and finally uses ReLU activation function for operation.

[0032] The specific steps are as follows:

[0033] S2.1 Input the spatial multi-view into the generator for multi-view encoding; the details are as follows:

[0034] S2.1.1 Preprocess the data input to the generator to obtain discrete and continuous features;

[0035] For discrete values, an embedding layer is used to project the discrete values ​​onto... Discrete features are obtained from a 3D vector space;

[0036] For continuous values, a linear layer is used to project the continuous values ​​onto... Continuous features are obtained in the 3D vector space;

[0037] S2.1.2. The discrete vector and the continuous vector are concatenated and then input into the encoder;

[0038] Specifically, global features Projection yields discrete global feature vectors and continuous global feature vectors, which are then concatenated along the dimensional direction (column direction) of the node and edge features of the region and the node and edge features of the point, respectively.

[0039] Discrete vector of points and point continuous vector The splicing process is as follows:

[0040]

[0041]

[0042] In the formula, , , , These are the node weight parameter, edge weight parameter, node offset parameter, and edge offset parameter, respectively. These are the node features of the spliced ​​points; These are the edge features of the points after splicing;

[0043] Node characteristics of the region Sum of edge features Obtained using the same splicing process;

[0044] The multi-view encoder takes the node features of the stitched region and the node features of the stitched point as input, and introduces the edge features of the stitched region and the edge features of the stitched point to represent the similarity between the nodes of the region and the point. An improved graph attention network is used for feature update.

[0045] S2.1.3 Calculate the attention correlation coefficient between nodes of the spliced ​​points using a mask-based attention mechanism;

[0046] Attention correlation coefficient The specific expression is as follows:

[0047]

[0048] In the formula, , , These are the first weight parameter, the node embedding weight parameter, and the edge embedding weight parameter, respectively. It is the LeakyReLU activation function. It's a splicing operation. This represents the embedding of node i after splicing at the k-th layer. This represents the edge embedding between node i and node j after splicing at the k-th layer;

[0049] S2.1.4. Use the softmax mechanism to regularize the attention coefficients of all neighbors of the stitched nodes to avoid overfitting; at the same time, use a multi-head attention mechanism to stabilize the learning process.

[0050] The specific expression for the regularized attention coefficient is as follows:

[0051]

[0052] In the formula, Indicates the area The neighborhood group, It is the time neighbor or spatial neighbor node of node i. If and only if ;

[0053] The specific expression for the stable learning process of multi-head attention mechanism is as follows:

[0054]

[0055] In the formula, It is the second weighting parameter. Y represents the embedding of node i at position k+1 after concatenation, and Y is the number of attention heads.

[0056] S2.1.5 In the last layer (k-th layer) of the encoder, the averaging operation is replaced with the concatenation operation, and the ReLU activation function is used, as shown in the following expression:

[0057]

[0058] In the formula, It is the second weighting parameter. It is the encoded representation of node i in the spatial point topology diagram;

[0059] Thus, the encoded representation of the spatial point topology map is obtained. The same method is used to obtain the encoded representation of the spatial region topology map. ;

[0060] S2.2, Input the encoder's output into the decoder to predict the delivery person's spatiotemporal behavior based on the spatial region topology map and spatial point topology map; specifically as follows:

[0061] S2.2.1 For the spatiotemporal behavior prediction of delivery personnel in the spatial region topology map, at each step of decoding, the decoder will calculate the probability of the node in each candidate region and select the node with the highest probability as the spatiotemporal behavior prediction output of the delivery personnel.

[0062] Specifically, the conditional probability calculated using the chain rule is used to describe the entire decoding process of the delivery person at the current time t, as shown in the following expression:

[0063]

[0064] In the formula, It is the region encoding representation output by the encoder; It involves analyzing the spatiotemporal behavior of delivery personnel within an already generated spatial region topology map. Obtained by aggregation; This indicates the delivery person's personalized behavioral preferences; This represents the learnable parameters in the decoder; It represents the spatiotemporal behavior of the delivery person at node i in the spatial region topology graph; m∈(1,25);

[0065] For any step r during decoding, the goal is to find the most likely neighbor output in the previous step (r-1), while ensuring that no node is output repeatedly. Then, the most likely neighbor selected in the current decoding step is used as the input for the next decoding step to obtain a new hidden state. ;

[0066] Specifically, the hidden state is in step r-1. and The vector concatenation is used as the query, and the neighbors of the output node in step r-1 are used as the key to calculate the probability distribution among all neighboring nodes. The expression is as follows:

[0067]

[0068] In the formula, , These are the sixth and seventh weight parameters, respectively. These are region-learnable weight parameters. This represents the encoded representation of the neighboring nodes output at step r-1 in the spatial region topology graph. This represents the hidden state at step r-1 in the spatial region topology graph, if node If there was no output in the previous steps, then ;

[0069] The probability of each output node in step r is obtained through softmax operation, as shown in the following expression:

[0070]

[0071] The maximum value of all probabilities in step r is taken to form the spatiotemporal behavior prediction output of the deliveryman in the spatial region topology map of step r. The expression is as follows:

[0072] ;

[0073] S2.2.2. The spatial region transfer pattern is used to predict the spatiotemporal behavior of delivery personnel in the spatial location topology map; the details are as follows:

[0074] Encoding representation of node i in the spatial point topology map Transfer pattern vectors with spatial regions Concatenate them as input Then calculate the probability distribution among all its neighboring nodes. The details are as follows:

[0075]

[0076]

[0077] In the formula, , These are the eighth and ninth weight parameters, respectively. These are learnable weight parameters for the points. This represents the encoded representation of node j, a neighbor of node i, in a spatial point topology graph. This represents the hidden state at step r-1 in the spatial point topology graph. If node j did not have any output in previous steps, then... ;

[0078] The probability of each output node in step r is obtained through softmax operation, as shown in the following expression:

[0079]

[0080] In the formula, It is an exponential function with the mathematical constant e as its base;

[0081] The maximum value of all probabilities in step r is taken to obtain the spatiotemporal behavior prediction output of the spatial point topology map in step r. The expression is as follows:

[0082]

[0083] In the formula, The function represents finding, among all possible values ​​of node j, the one that satisfies Take the maximum value of j;

[0084] S2.3 The discriminator takes the predicted spatiotemporal behavior and the actual spatiotemporal behavior as inputs and distinguishes them according to the reward value of the corresponding spatiotemporal behavior;

[0085] The goal of the discriminator is to distinguish between predicted spatiotemporal behavior and actual spatiotemporal behavior, as follows:

[0086] S2.3.1, By incorporating delivery personnel's personalized behavioral preferences into the discriminator. Simplify model training; incorporate delivery personnel's personalized behavioral preferences. The expression for the discriminator is:

[0087]

[0088] In the formula, It is the state of the current spatiotemporal behavior node; It is the action choice of the current spatiotemporal behavior; For the reward estimator, the expression is as follows:

[0089]

[0090] In the formula, It is a discount factor. Given the state s and action of the current spatiotemporal behavior node The state of the next spatiotemporal behavior node, It is a reward approximator. It is a reward-shaping item;

[0091] It is a reward approximator; specifically, it represents the characteristics of the delivery person's spatiotemporal behavior as a unique encoded vector. It is then concatenated with the vectors learned by the embedding layer of the convolutional neural network, and a two-layer feedforward neural network is used to learn the concatenated vectors to obtain... ;

[0092] It is the reward shaping term for the current spatiotemporal behavior node's state s, and the input is the feature of the deliveryman's spatiotemporal behavior. ,in, It is the feature vector of the deliveryman's spatiotemporal behavior in the current spatiotemporal behavior node state s. It is a vector representing the delivery person's personalized behavioral preferences, obtained using a two-layer feedforward neural network. ; It is the reward shaping term for state s+1, and the input is the characteristics of the deliveryman's spatiotemporal behavior. ,in, It is the feature vector of the deliveryman's spatiotemporal behavior in state s+1;

[0093] S2.3.2. Set the simplified reward value as the policy objective of entropy regularization to obtain the deliveryman's expected return; the expression is as follows:

[0094]

[0095] In the formula, It is a logarithmic function with the mathematical constant e as the base. It is the state s and action a of the current spatiotemporal behavior node in the deliveryman's personalized behavioral preferences. The immediate expected return;

[0096] S2.3.3, For real-time spatiotemporal behavioral data and predictive spatiotemporal behavior data The model parameters are updated using the batch gradient descent algorithm;

[0097] In this example, the delivery person's spatiotemporal behavior is segmented into M state-action pairs (M∈(0,N), N is a natural integer), forming a training set denoted as . During the training process, in each round p from Randomly select a batch of real spatiotemporal behavioral data , p∈(1, 100);

[0098] S2.4 Output results through the value estimator ;

[0099] Value estimator Used to calculate the state s of the current spatiotemporal behavior node in the deliveryman's personalized behavioral preferences The cumulative expected return is determined by the characteristics of the delivery driver's spatiotemporal behavior. The output is the expected reward for the deliveryman's spatiotemporal behavior;

[0100] S3. Update the generator, discriminator and value estimator respectively to obtain the optimal value;

[0101] The generator update is trained using a near-end policy optimization algorithm, expressed as follows:

[0102]

[0103] In the formula, This represents the probability ratio between the old and new strategies. These are the cropping parameters. It is a clipping function. It is the dominance estimation function; for a deliveryman's spatiotemporal behavior data containing T time points, the specific expression of the dominance estimation function is as follows:

[0104]

[0105] in,

[0106] In the formula, It is a discount factor, parameter Used to balance bias and variance. Let be the expected reward of the delivery driver's spatiotemporal behavior at time t. It is the estimated value of the value estimator;

[0107] The update expression for the discriminator is:

[0108]

[0109] In the formula, The output of the discriminator represents the state s, action a, and the delivery person's personalized behavioral preferences at the current spatiotemporal behavior node. The discriminator determines the probability that the data is real spatiotemporal behavioral data. This represents the expected value of real spatiotemporal behavioral data. This represents the expected value of the predicted spatiotemporal behavior data;

[0110] The value estimator update expression is:

[0111]

[0112] In the formula, This represents the reward value based on the deliveryman's personalized behavioral preferences u, the current spatiotemporal behavior node's state s, and action a, and is used as... and The sum of, , These represent the new and old parameters of the value estimator, respectively;

[0113] The optimal value was finally obtained; the details are as follows:

[0114] Based on the update process of the generator, discriminator and value estimator, the three models compete against each other during training, prompting them to continuously improve until they reach the optimal state; the three models use the training set to optimize and update the model parameters, use the validation set to select parameters, and use the test set to evaluate the generalization performance of the model.

[0115] The goal of the generator update process is to find the spatiotemporal behavioral strategy that produces the maximum reward;

[0116] The goal of the discriminator update process is to provide a reward function that can distinguish between real spatiotemporal behavior and predicted spatiotemporal behavior based on the reward value;

[0117] The goal of the value estimator update process is to provide the generator with value information about the current spatiotemporal state, thereby helping the generator to more effectively optimize its spatiotemporal behavior strategy.

[0118] Beneficial effects of the present invention

[0119] 1. This invention introduces spatial multi-view and delivery personnel's personalized behavioral preferences into an adversarial reinforcement learning network, which can recover the reward function from the delivery personnel's historical spatiotemporal behavior data. This can effectively avoid the problem of difficulty in designing reward functions in reinforcement learning, and at the same time learn optimized reward and policy functions, thereby improving the interpretability of the model and the accuracy of prediction.

[0120] 2. This invention introduces multiple spatial views, including a spatial region topology map and a spatial point topology map. The spatial region transfer pattern is obtained by predicting the spatiotemporal behavior of delivery personnel through the spatial region topology map. The spatial region transfer pattern guides the process of predicting the spatiotemporal behavior of delivery personnel through the spatial point topology map, thereby improving the prediction accuracy.

[0121] 3. In the process of predicting the spatiotemporal behavior of delivery personnel, this invention takes into account the delivery personnel's selection preferences in different locations, introduces the delivery personnel's personalized behavioral preferences, reduces the large search space in the spatiotemporal behavior prediction, speeds up the calculation efficiency, and improves the accuracy of the spatiotemporal behavior prediction of delivery personnel. Attached Figure Description

[0122] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0123] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0124] This embodiment uses the publicly available LaDe logistics delivery dataset, which is the first comprehensive last-mile delivery dataset collected by Cainiao Network (one of China's largest logistics companies), and it has three significant characteristics:

[0125] It was a large-scale operation, involving 21,000 delivery workers and 10,677,000 packages delivered over six months of actual operation.

[0126] The information is comprehensive. The dataset provides raw package information, such as the location and time requirements of the package, as well as task event information, recording the time and location of events such as task acceptance and task completion by the delivery person.

[0127] The dataset is diverse, containing data from various scenarios such as parcel pickup and delivery, as well as data from multiple cities in China (Shanghai, Chongqing, Hangzhou). Each city has unique spatiotemporal patterns due to its unique characteristics such as population, economy, and geographical location.

[0128] This dataset uses the LaDe logistics dataset, specifically a real-world urban parcel pickup dataset from Shanghai. The Shanghai dataset contains pickup records from 4,502 delivery personnel in Shanghai, China, spanning six months from May to October 2023. In the dataset, the training, validation, and test sets are partitioned chronologically in a 6:2:2 ratio. Detailed information about the dataset is shown in Table 1 below.

[0129] Table 1 Dataset Details

[0130] City time Number of people Spatiotemporal behavior number Average arrival time Average number of items picked up Shanghai 6 months 4502 people 96,000 151 minutes 15 items

[0131] See Figure 1 A spatiotemporal behavior prediction method based on multi-view personalized inverse reinforcement learning includes:

[0132] S1. Extract spatial multi-views from the original deliveryman spatiotemporal behavior data. The spatial multi-views include: spatial region topology map and spatial point topology map.

[0133] Specifically, based on the LaDe logistics delivery dataset, spatiotemporal behavior data for each delivery person in Shanghai, China, from May to October 2023 were obtained, including: spatiotemporal constraints of uncollected tasks (including: package location and promised pickup time), characteristics of the most recent pickup tasks (including: pickup time and pickup location), and characteristics of the delivery person (including: total working days and average number of pickups per day).

[0134] Furthermore, the original spatiotemporal behavior data of delivery personnel is processed, including: processing of delivery personnel's own attribute characteristics (attendance rate, vehicle speed, working hours), processing of delivery task characteristics (geographical location information, location information, distance information, deadline information), limiting the length of spatiotemporal behavior (i.e., unifying the data length), and processing of dataset partitioning (the training set, validation set, and test set are divided in a 6:2:2 ratio).

[0135] In the processed spatiotemporal behavior data of delivery personnel, both regions and locations are treated as nodes, from which spatial multi-views are extracted;

[0136] Spatial multi-view includes: spatial region topology map Spatial point topology map ;in, This includes: the set of points in the region, the set of edges in the region, the node features of the region, and the edge features of the region; This includes: the set of points, the set of edges of points, the node features of points, and the edge features of points;

[0137] Node characteristics of the point The specific definitions are as follows:

[0138]

[0139] In the formula, It is the geographic location information (longitude and latitude) related to the delivery task. and The area where the mission is located Number and function type (school, commercial area, factory area and residential area). It is the distance between the delivery person and the location of the delivery task. Representing the current moment, and These are the time when the delivery person accepts the delivery task and the deadline for the task. It is the final, actual completion time of the task;

[0140] Node characteristics of the region The expression is as follows:

[0141]

[0142] In the formula, It is the central geographical location information (longitude and latitude) of the region. and It is a region Number and function type (school, commercial area, factory area and residential area). It is the distance between the delivery person and the location of the delivery task. Representing the current moment, This is the earliest deadline for all delivery tasks in the region;

[0143] Edge features of the point The specific definitions are as follows:

[0144] (i≠j, i, j∈W)

[0145] In the formula, Represents a node and nodes Euclidean distance at corresponding positions Represents a node and nodes The difference between the corresponding task deadlines, Represents a node and nodes The connectivity between corresponding positions, the The definition is as follows:

[0146]

[0147] Edge features of the region The specific definitions are as follows:

[0148] (I≠J, I, J∈Z)

[0149] In the formula, Indicates the region corresponding to node i and the region corresponding to node j Euclidean distance corresponding to the center point position, Indicates the area and region The difference between the earliest deadlines of the tasks. Indicates the area and region Connectivity between corresponding center point locations;

[0150] In addition to location and area characteristics, delivery personnel's own attributes (such as attendance rate) Speed Working hours ),weather or weekdays Characteristics also influence the delivery person's spatiotemporal behavior;

[0151] The delivery person's own attributes, weather conditions, and workday characteristics are collectively referred to as global characteristics. global features The definition is as follows:

[0152] ;

[0153] S2. Input the spatial multi-view into the personalized adversarial inverse reinforcement learning network to obtain the deliveryman's predicted spatiotemporal behavior, the deliveryman's actual spatiotemporal behavior, the reward value of the predicted spatiotemporal behavior, and the expected reward of the deliveryman's spatiotemporal behavior corresponding to the state.

[0154] Personalized adversarial inverse reinforcement learning networks include: a generator, a discriminator, and a value estimator;

[0155] The generator includes: a multi-view encoder and a decoder;

[0156] The multi-view encoder adopts an improved graph attention network based on the traditional graph attention network, which introduces mask attention mechanism, regularization and mean processing of multi-head attention, uses averaging operation to replace concatenation operation, and finally uses ReLU activation function for operation.

[0157] The specific steps are as follows:

[0158] S2.1 Input the spatial multi-view into the generator for multi-view encoding, as follows:

[0159] S2.1.1 Preprocess the data input to the generator to obtain discrete and continuous features;

[0160] For discrete values, an embedding layer is used to project the discrete values ​​onto... Discrete features are obtained from a 3D vector space;

[0161] For continuous values, a linear layer is used to project the continuous values ​​onto... Continuous features are obtained in the 3D vector space;

[0162] In this embodiment, the discrete values ​​include: region ID number, function type, delivery person attendance rate, delivery person vehicle speed, delivery person working hours, weather characteristics, and workday characteristics. An embedding layer projects these discrete values ​​onto... Discrete features are obtained from a 3D vector space;

[0163] Continuous values ​​include: longitude and latitude information, distance information between the delivery person and the delivery task location, current time information, the time the delivery person accepted the delivery task, the deadline for the delivery task, and the final actual completion time of the delivery task. A linear layer is used to project these continuous values ​​onto... Continuous features are obtained in the 3D vector space;

[0164] S2.1.2. The discrete vector and the continuous vector are concatenated and then input into the encoder;

[0165] global features Projection yields discrete global feature vectors and continuous global feature vectors, which are then concatenated along the dimensional direction (column direction) of the node and edge features of the region and the node and edge features of the point, respectively.

[0166] Discrete vector of points and point continuous vector The splicing process is as follows:

[0167]

[0168]

[0169] In the formula, , , , These are the node weight parameter, edge weight parameter, node offset parameter, and edge offset parameter, respectively. These are the node features of the spliced ​​points; These are the edge features of the points after concatenation; the model is trained using the backpropagation algorithm. , , , Optimized during training;

[0170] Node characteristics of the region Sum of edge features The same stitching process is used to obtain the features; the multi-view encoder takes the node features of the stitched region and the node features of the stitched point as input, and introduces the edge features of the stitched region and the edge features of the stitched point to represent the similarity between the nodes of the region and the point, and uses an improved graph attention network for feature update.

[0171] S2.1.3 Calculate the attention correlation coefficient between nodes of the spliced ​​points using a mask-based attention mechanism;

[0172] The key to the calculation process is to calculate the attention correlation coefficient between nodes. The specific expression is as follows:

[0173]

[0174] In the formula, , , These are the first weight parameter, the node embedding weight parameter, and the edge embedding weight parameter, respectively. It is the LeakyReLU activation function. It's a splicing operation. This represents the embedding of node i after splicing at the k-th layer. This represents the edge embedding between nodes i and j after concatenation at the k-th layer; the model is trained using the backpropagation algorithm. , , Optimized during training;

[0175] S2.1.4. The attention coefficients of all neighbors of the concatenated nodes are regularized using a softmax mechanism to avoid overfitting; simultaneously, a multi-head attention mechanism is used to stabilize the learning process; the specific expression for the regularized attention coefficients is as follows:

[0176]

[0177] In the formula, Indicates the area The neighborhood group, It is the time neighbor or spatial neighbor node of node i. If and only if ;

[0178] To stabilize the learning process of the attention mechanism, a multi-head attention mechanism is introduced, using Y attention heads and regularized attention coefficients. To update, in this embodiment, 8 attention heads are used, and the specific expressions are as follows:

[0179]

[0180] In the formula, It is the second weighting parameter. This represents the embedding of node i at position k+1 after concatenation; the model is trained using the backpropagation algorithm. Optimized during training;

[0181] The edge representation of the concatenated points is updated using a linear transformation, as shown in the following expression:

[0182]

[0183] In the formula, , , These are the third, fourth, and fifth weighting parameters, respectively. This represents the edge embedding of the spliced ​​node i and spliced ​​node j at the (k+1)th layer. This represents the embedding of node j at the k-th layer after splicing; This represents the edge embedding between the concatenated nodes i and j at the k-th layer; the model is trained using the backpropagation algorithm. , , The result is obtained through optimization during training; in this embodiment, k is 3.

[0184] S2.1.5 In the last layer of the encoder, the averaging operation is replaced with the concatenation operation, and the ReLU activation function is used, as shown in the following expression:

[0185]

[0186] In the formula, It is the second weighting parameter. It is the encoded representation of node i in the spatial point topology diagram;

[0187] Thus, the encoded representation of the spatial point topology map is obtained. The same method is used to obtain the encoded representation of the spatial region topology map. ;

[0188] S2.2 Input the encoder's output into the decoder to complete the spatiotemporal behavior prediction of delivery personnel based on the spatial region topology map and spatial point topology map;

[0189] The encoder obtains the following codes: the code for the spatial point topology map and the code for the spatial region topology map;

[0190] Among them, the prediction results of delivery personnel's spatiotemporal behavior from the spatial region topology map can serve as important guiding information for the prediction of delivery personnel's spatiotemporal behavior from the spatial point topology map; a masked attention mechanism based on a long short-term memory network is designed to simulate the spatiotemporal behavior prediction process of delivery personnel, as detailed below:

[0191] S2.2.1 For the spatiotemporal behavior prediction of delivery personnel in the spatial region topology map, at each step of decoding, the decoder calculates the probability of nodes in each candidate region and selects the node with the highest probability as the output of the delivery personnel spatiotemporal behavior prediction, as follows:

[0192] The conditional probability calculated using the chain rule is used to describe the entire decoding process of the delivery person at the current time t. The specific expression is as follows:

[0193]

[0194] In the formula, It is the region encoding representation output by the encoder; It involves analyzing the spatiotemporal behavior of delivery personnel within an already generated spatial region topology map. Obtained by aggregation; This indicates the delivery person's personalized behavioral preferences; This represents the learnable parameters in the decoder; It represents the spatiotemporal behavior of the delivery person at node i in the spatial region topology graph; m∈(1,25);

[0195] For any step r, r∈m, during decoding, the goal is to find the most probable neighbor output in the previous step (r-1), while ensuring that no node is output repeatedly. Then, the most probable neighbor selected in the current decoding step is used as the input for the next decoding step, resulting in a new hidden state. Specifically, the hidden state is determined in step r-1. and The concatenation of the values ​​is used as the query, and the neighbors of the node output in the previous step are used as the key to calculate the probability distribution among all neighboring nodes. The details are as follows:

[0196]

[0197] In the formula, , These are the sixth and seventh weight parameters, respectively. These are region-learnable weight parameters. This represents the encoded representation of the neighboring nodes output at step r-1 in the spatial region topology graph. This represents the hidden state at step r-1 in the spatial region topology graph, if node If there is no output in the previous steps, it means that... The model is trained using the backpropagation algorithm. , , Optimized during training;

[0198] The probability of each output node in step r is obtained through softmax operation, as shown in the following expression:

[0199]

[0200] The maximum value of all probabilities in step r is taken to form the spatiotemporal behavior prediction output of the deliveryman in the spatial region topology map of step r, as shown in the following expression:

[0201] ;

[0202] S2.2.2. The spatial region transfer pattern is used to predict the spatiotemporal behavior of delivery personnel in the spatial location topology map; the details are as follows:

[0203] The spatiotemporal behavior prediction of delivery personnel based on a spatial point topology map incorporates spatial region transition patterns in the r-th decoding step to calculate the conditional probability of each point. Specifically, the encoded representation of node i in the spatial point topology map... Transfer pattern vectors with spatial regions Concatenate them as input Then calculate the probability distribution among all its neighboring nodes. The details are as follows:

[0204]

[0205]

[0206] In the formula, , These are the eighth and ninth weight parameters, respectively. These are learnable weight parameters for the points. This represents the encoded representation of node j, a neighbor of node i, in a spatial point topology graph. This represents the hidden state at step r-1 in the spatial point topology graph. If node j has not been output in previous steps, it is represented as... The model is trained using the backpropagation algorithm. , , Optimized during training;

[0207] The probability of each output node in step r is obtained through softmax operation, as shown in the following expression:

[0208]

[0209] In the formula, It is an exponential function with the mathematical constant e as its base;

[0210] The maximum value of all probabilities in step r is taken to obtain the spatiotemporal behavior prediction output of the spatial point topology map in step r. The expression is as follows:

[0211]

[0212] In the formula, The function represents finding, among all possible values ​​of node j, the one that satisfies Take the maximum value of j;

[0213] S2.3 The discriminator takes the predicted spatiotemporal behavior and the actual spatiotemporal behavior as inputs and distinguishes them according to the reward value of the corresponding spatiotemporal behavior;

[0214] The goal of the discriminator is to distinguish between predicted spatiotemporal behavior and actual spatiotemporal behavior, as follows:

[0215] S2.3.1, By incorporating delivery personnel's personalized behavioral preferences into the discriminator. Simplify model training; incorporate delivery personnel's personalized behavioral preferences. The expression for the discriminator is:

[0216]

[0217] In the formula, It is the state of the current spatiotemporal behavior node; It is the action choice of the current spatiotemporal behavior; For the reward estimator, the expression is as follows:

[0218]

[0219] In the formula, It is a discount factor. ∈[0,1], It is the state s and action of the current spatiotemporal behavior node. The state of the next spatiotemporal behavior node, It is a reward approximator. It is a reward-shaping item;

[0220] It is a reward approximator; specifically, it represents the characteristics of the delivery person's spatiotemporal behavior as a unique encoded vector. It is then concatenated with the vectors learned by the embedding layer of the convolutional neural network, and a two-layer feedforward neural network is used to learn the concatenated vectors to obtain... ;

[0221] It is the reward shaping term for the current spatiotemporal behavior node's state s, and the input is the feature of the deliveryman's spatiotemporal behavior. ,in, It is the feature vector of the deliveryman's spatiotemporal behavior in the current spatiotemporal behavior node state s. It is a vector representing the delivery person's personalized behavioral preferences, obtained using a two-layer feedforward neural network. ; It is the reward shaping term for state s+1, and the input is the characteristics of the deliveryman's spatiotemporal behavior. ,in, It is the feature vector of the deliveryman's spatiotemporal behavior in state s+1;

[0222] S2.3.2. Set the differentiated reward value as the strategy objective of entropy regularization to obtain the deliveryman's expected return; the expression is as follows:

[0223]

[0224] In the formula, It is a logarithmic function with the mathematical constant e as the base. It is the state s and action a of the current spatiotemporal behavior node in the deliveryman's personalized behavioral preferences. The immediate expected return;

[0225] S2.3.3, For real-time spatiotemporal behavioral data and predictive spatiotemporal behavior data The model parameters are updated using the batch gradient descent algorithm;

[0226] In this example, the delivery person's spatiotemporal behavior is segmented into M state-action pairs (M∈(0,N), N is a natural integer), forming a training set denoted as . During the training process, in each round p from Randomly select a batch of real spatiotemporal behavioral data , p∈(1, 100);

[0227] S2.4 Output results through the value estimator ;

[0228] Value estimator Used to calculate the state s of the current spatiotemporal behavior node in the deliveryman's personalized behavioral preferences The cumulative expected return is determined by the characteristics of the delivery driver's spatiotemporal behavior. The output is the expected reward for the deliveryman's spatiotemporal behavior;

[0229] S3. Update the generator, discriminator and value estimator respectively to obtain the optimal value;

[0230] The generator update is trained using a near-end policy optimization algorithm, expressed as follows:

[0231]

[0232] In the formula, This represents the probability ratio between the old and new strategies. These are the cropping parameters. ∈ (0.1, 0.3), It is a clipping function. It is the dominance estimation function; for a deliveryman's spatiotemporal behavior data containing T time points, T∈[0,120], the specific expression of the dominance estimation function is as follows:

[0233]

[0234] in,

[0235] In the formula, It is a discount factor, parameter Used to balance bias and variance. Let be the expected reward of the delivery driver's spatiotemporal behavior at time t. It is the estimated value of the value estimator; T i ∈T;

[0236] The update expression for the discriminator is:

[0237]

[0238] In the formula, The output of the discriminator represents the state s, action a, and the delivery person's personalized behavioral preferences at the current spatiotemporal behavior node. The discriminator determines the probability that the data is real spatiotemporal behavioral data. This represents the expected value of real spatiotemporal behavioral data. This represents the expected value of the predicted spatiotemporal behavior data;

[0239] The value estimator update expression is:

[0240]

[0241] In the formula, This represents the reward value based on the deliveryman's personalized behavioral preferences u, the current spatiotemporal behavior node's state s, and action a, and is used as... and The sum of, , These represent the new and old parameters of the value estimator, respectively;

[0242] The optimal value was finally obtained; the details are as follows:

[0243] Based on the update process of the generator, discriminator and value estimator, the three models compete against each other during training, prompting them to continuously improve until they reach the optimal state; the three models use the training set to optimize and update the model parameters, use the validation set to select parameters, and use the test set to evaluate the generalization performance of the model.

[0244] The goal of the generator update process is to find the spatiotemporal behavioral strategy that produces the maximum reward;

[0245] The goal of the discriminator update process is to provide a reward function that can distinguish between real spatiotemporal behavior and predicted spatiotemporal behavior based on the reward value;

[0246] The goal of the value estimator update process is to provide the generator with value information about the current spatiotemporal state, thereby helping the generator to more effectively optimize its spatiotemporal behavior strategy.

[0247] The proposed spatial multi-view personalized adversarial inverse reinforcement learning method for predicting delivery personnel spatiotemporal behavior has been compared with mainstream methods in the field of delivery personnel spatiotemporal behavior prediction on the same Shanghai dataset. The results are shown in Table 1.

[0248] Table 1. Results of Effect Comparison

[0249] method HR@1 ACC@3 KRC LSD ED DeepRoute 55.53 30.04 55.91 2.87 1.78 Graph2Route 54.54 28.23 54.32 3.10 1.79 RankETPA 55.62 30.00 56.05 2.89 1.78 DRL4Route 56.31 30.12 56.14 2.79 1.78 This method 57.34 31.14 57.04 2.69 1.75 promote 1.8% 3.4% 1.6% 3.6% 1.7%

[0250] In the table, HR@1 is the hit rate, ACC@3 is the accuracy, KRC is the Kendall correlation coefficient, LSD is the position squared deviation, and ED is the edit distance;

[0251] Among them, the higher the HR@1, ACC@3, and KRC, and the lower the LSD and ED, the better the spatiotemporal behavior prediction performance of the method. It can be seen that the present method is better than the current best method in all indicators, and improves the indicator effect by 1.6% to 3.6%, which proves the effectiveness of the present method.

[0252] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the scope of protection of the appended claims.

Claims

1. A spatio-temporal behavior prediction method based on multi-view personalized inverse reinforcement learning, characterized in that, The method comprises the following steps: S1, extracting a spatial multi-view from original courier spatio-temporal behavior data; The spatial multi-view comprises a spatial region topology graph and a spatial point topology graph; S2, inputting the spatial multi-view into an individualized adversarial inverse reinforcement learning network to obtain a predicted spatio-temporal behavior of the courier, a real spatio-temporal behavior of the courier, a reward value of the predicted spatio-temporal behavior and an expected return of a state corresponding to the spatio-temporal behavior of the courier; The individualized adversarial inverse reinforcement learning network comprises a generator, a discriminator and a value estimator; The generator comprises a multi-view encoder and a decoder; The multi-view encoder is based on a traditional graph attention network, and a mask attention mechanism, regularization of multi-head attention and mean processing are introduced; and an average operation is used instead of a splicing operation; finally, a ReLU activation function is used; S3, updating the generator, the discriminator and the value estimator respectively to obtain an optimal value; The updating of the generator, the discriminator and the value estimator is specifically as follows: The updating of the generator is trained by using a proximal policy optimization algorithm, and an expression is as follows: ； wherein denotes the probability ratio of new versus old policy; is a clipping parameter; is a clipping function; is a advantage estimation function, expressed as follows: ； wherein ; wherein is a discount factor, the parameter for balancing bias and variance, is the expected return of the courier's spatio-temporal behavior at time t, is the estimated value of the value estimator; An expression for updating the discriminator is as follows: ； wherein represents the output of the discriminator, in the state s of the current spatio-temporal action node, the action a and the personalized behavioral preference of the delivery person represents the probability that the discriminator judges the data to be real spatio-temporal behavior data, represents the expected value of the real spatio-temporal behavior data, represents the expected value of the predicted spatio-temporal behavior data; An expression for updating the value estimator is as follows: ； In the formula, represents the reward value based on the delivery person's personalized behavior preference u, the state s of the current spatio-temporal behavior node and the action a, and takes it as The sum of and, , respectively represent the new parameter and the old parameter of the value estimator.

2. The spatio-temporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 1, characterized in that: The spatial region topology graph comprises a point set of a region, an edge set of the region, node features of the region and edge features of the region; and the spatial point topology graph comprises a point set, an edge set of the point, node features of the point and edge features of the point; An expression of the node features of the region is as follows: ； In the formula, is the central geographic position information of the region, and is the number and function type, is the distance between the deliveryman and the location where the delivery task is located, represents the current time, is the earliest deadline of all delivery tasks in the region; The edge features of the region are specifically defined as follows: (I≠J, I, J e Z); wherein, denotes the region corresponding to node i and the region corresponding to node j the Euclidean distance between the center point positions, denotes the region and the region the difference of the earliest deadline of tasks in the region denotes the region and the region the connectivity between the center point positions; The node features of the point are specifically defined as follows: ； In the formula, is the geographic location information related to the delivery task, and is the number and function type of the area where the task is located, is the distance between the delivery personnel and the location where the delivery task is located, represents the current time, and are the time when the delivery personnel accepts the delivery task and the deadline of the task, respectively, is the final actual completion time of the task; The edge features of the point are specifically defined as follows: (i≠j, i, j e W); wherein, denotes a node and a node the Euclidean distance between the corresponding locations, denotes a node and a node the difference between the corresponding task deadlines, denotes a node and a node the connectivity between the corresponding locations, said is defined as follows: 。 3. The spatio-temporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 1, characterized in that: The steps of S2 are as follows: S2.1, inputting the spatial multi-view into the generator for multi-view encoding; S2.2, inputting the multi-view encoding into the decoder to complete courier spatio-temporal behavior prediction based on the spatial region topology graph and the spatial point topology graph; S2.3, the discriminator taking the predicted spatio-temporal behavior and the real spatio-temporal behavior as inputs, and distinguishing them according to the reward value of the corresponding spatio-temporal behavior, comprising the following steps: S2.3.1, by incorporating the individualized behavior preferences of the delivery person in the discriminator Simplifies model training; S2.3.2, setting the simplified reward value as an entropy regularization strategy target to obtain the expected return of the courier; S2.3.3, for real spatiotemporal behavior data and predicted spatiotemporal behavior data updating the model parameters using a batch gradient descent algorithm; S2.4, outputting a result by the value estimator.

4. The spatio-temporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 3, characterized in that: The steps of inputting the spatial multi-view into the generator for multi-view encoding are as follows: S2.1.1, pre-processing data input into the generator to obtain discrete features and continuous features; For discrete values, the discrete values are projected to a continuous vector space using an embedding layer to obtain discrete features; For continuous values, the continuous values are projected to a discrete set of values using a linear layer a continuous feature is obtained; S2.1.2, splicing the discrete vector and the continuous vector and inputting them into the encoder; S2.1.3, calculating a node-to-node attention correlation coefficient of the spliced point by using a mask attention mechanism; Attention-related coefficients The specific expression of the attention-related coefficients is as follows: ； wherein, , , are first weight parameters, node embedding weight parameters, edge embedding weight parameters, is a LeakyReLU activation function, is a concatenation operation, denotes the embedding of the point node i after concatenation at the k-th layer, denotes the edge embedding of the point node i and the node j after concatenation at the k-th layer. S2.1.4, using a softmax mechanism to regularize the attention coefficients of all neighbors of the spliced point node to avoid overfitting; and using a multi-head attention mechanism to stabilize the learning process; S2.1.5, using an average operation instead of a splicing operation in the last layer of the encoder, and using a ReLU activation function.

5. The spatio-temporal behavior prediction method based on multi-view personalized inverse reinforcement learning according to claim 3, characterized in that: The steps of inputting the multi-view encoding into the decoder to complete courier spatio-temporal behavior prediction based on the spatial region topology graph and the spatial point topology graph are as follows: S2.2.1, for the courier space-time behavior prediction of the space region topology graph, at each decoding step, the decoder calculates the probability of each candidate region node, and selects the node with the maximum probability as the courier space-time behavior prediction output; The expression is as follows: ； In the formula, is the probability of each output node in the rth step of the spatial region topology graph obtained by the softmax operation; The function represents finding j that makes the maximum value among all possible values of node j. S2.2.2, in the courier space-time behavior prediction of the space point topology graph, the transfer mode of the space region is used to obtain the courier space-time behavior prediction of the space point topology graph; The expression is as follows: ； In the formula, is the probability of each output node in the rth step of the spatial point topology graph obtained by the softmax operation; The function represents finding j that makes the maximum value among all possible values of node j.

Images