A transportation track generation method and device, a terminal and a storage medium

Abstract

The application relates to the technical field of computers and discloses a transportation track generation method and device, equipment and a storage medium, wherein GPS track data is obtained by acquiring road GPS point data and carrying out noise filtering processing; a state-action value function, a strategy function and a discrimination function are established based on the GPS track data and five elements of a Markov decision process; parameters of the state-action value function, the strategy function and the discrimination function are updated to construct a generated imitation reinforcement learning model; a meta-learning algorithm is used to train the generated imitation reinforcement learning model, and a transportation track is generated according to task-specific parameters obtained by training the meta parameters; the transportation track generation method and device, the terminal and the storage medium provided by the application can construct the generated imitation reinforcement learning model, improve the accuracy of the model, generate a city transportation track, and thus adapt to a complex travel environment.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, terminal and storage medium for generating transportation trajectories. Background Technology

[0002] With urbanization, urban transportation issues have become increasingly important. The generation of big data and the development of computer technology allow for the analysis of transportation trajectories on the existing urban road network. This helps to understand urban traffic patterns, make reasonable predictions, and effectively plan infrastructure. It also helps alleviate the burden of data collection, eliminate user privacy concerns, and provides effective data sources and guidance for prediction and recommendation services in other cities.

[0003] Currently, existing technologies employ two main approaches. One approach uses common machine learning models to simply mimic historical travel trajectories, such as using transition probability matrices, RNNs, and LSTMs to predict the next location. However, this method ignores the expected reward for future trajectory information and struggles to adapt to complex travel environments. Another approach models the trajectory generation problem as a Markov sequence decision problem, maximizing the expected reward by making action decisions at different intersections through imitation reinforcement learning. Common imitation learning algorithms, such as IRL, maximum entropy IRL, and dynamic environmental interaction, are used to generate travel trajectories. However, because user travel strategies are often highly complex, parameterized modeling of these strategies often fails to approximate the true distribution, resulting in models that are ineffective at generating urban travel trajectories and exhibit poor accuracy. Summary of the Invention

[0004] This invention provides a method, apparatus, terminal, and storage medium for generating transportation trajectories, which can construct a generative reinforcement learning model to generate urban transportation trajectories, adapting to complex travel environments and improving the accuracy of the model.

[0005] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a method for generating a transportation trajectory, comprising the following steps:

[0006] Obtain road GPS point data and perform noise filtering to obtain GPS trajectory data;

[0007] Based on the GPS trajectory data and the state set and action set in the five elements of the Markov decision process, a state-action value function, a generation policy function, and a discrimination function are established. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm. The parameters of the state-action value function, the generation policy function, and the discrimination function are updated to construct a generative imitation reinforcement learning model.

[0008] A meta-learning algorithm is adopted, in which the state-action value function, the generation policy function and the discrimination function are used as meta-parameters in the meta-learning algorithm. The meta-parameters are subjected to meta-iterative training, thereby training the generative imitation reinforcement learning model, and the transportation trajectory is generated based on the specific parameters of the task obtained from the training of the meta-parameters.

[0009] Furthermore, based on the GPS trajectory data and the state set and action set from the five elements of the Markov decision process, a state-action value function, a generation policy function, and a discrimination function are established. The discrimination function is used to identify the generated trajectory as a real trajectory using a minimax game algorithm. The parameters of the state-action value function, generation policy function, and discrimination function are updated to construct a generative imitation reinforcement learning model, specifically including:

[0010] The traffic flow signal of the road is acquired, and the traffic flow signal is incorporated into the GPS trajectory data as shared external information to obtain new GPS trajectory data;

[0011] Based on the new GPS trajectory data and the state set and action set in the five elements of the Markov decision process, a state-action value function, a generation strategy function, and a discrimination function are established. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm.

[0012] The state-action value function, the generation policy function, and the discrimination function are parameterized, and the state-action value function and the generation policy function are updated through joint training. Their expressions are as follows:

[0013]

[0014] in, The loss function is the state-action value function; The loss function is the generation strategy function; c is the weight coefficient.

[0015] The parameter update process of the state-action value function conforms to the Q-learning algorithm as follows:

[0016]

[0017] in, Let E be the loss function of the state-action value function; E be the expected reward; Q be the loss function of the state-action value function. ψ (s t ,a t R(s) is the state-action value function at the current moment; t ,a t R(s) is the reward function. t ,a t )=-logDφ (s t ,a t ), D φ (s t ,a t ) represents the discrimination function at the current moment; γ is the discount factor used to calculate the expected reward; π θ (a t+1 |s t+1 Q is the generation policy function for the next time step; ψ (s t+1 ,a t+1 ) is the state-action value function for the next time step; a t+1 This represents the action in the next moment;

[0018] The parameter update process of the generation policy function conforms to the gradient descent algorithm as follows:

[0019]

[0020] in, The loss function of the generation policy function; E is the expected reward; Q ψ (s t ,a t ) represents the state-action value function at the current moment; π θ (a t |s t H(π) is the generation strategy function at the current time; μ is a user-defined parameter; H(π) is the generation strategy function at the current time. θ ) represents the regularization term in the parameter update process of the generation strategy function;

[0021] The discrimination function is updated with parameters until it converges to the point where it is difficult to distinguish between the generated trajectory and the real trajectory, thereby constructing a generative imitation reinforcement learning model. The parameter update process of the discrimination function is as follows:

[0022]

[0023] in, E is the loss function of the discrimination function; π The expected reward for the generation strategy; The expected reward for the expert strategy; D(s) t ,a t ) is the discrimination function at the current moment.

[0024] Furthermore, the meta-learning algorithm is employed, using the state-action value function, generation policy function, and discrimination function as meta-parameters in the meta-learning algorithm. These meta-parameters undergo meta-iterative training to train the generative imitation reinforcement learning model. The transportation trajectory is generated based on the task-specific parameters obtained from the training of these meta-parameters, specifically including:

[0025] The area to be generated for transportation trajectories is divided into multiple blocks, and the blocks are spatially adjacent and have similar road network structures to ensure the similarity of the traffic flow signals. Based on the task division criteria of time and geographic similarity, the blocks and GPS trajectory data of the blocks at different time periods are defined as subtasks of the meta-learning algorithm.

[0026] The state-action value function, the generation policy function, and the discrimination function are used as meta-parameters of the sub-task;

[0027] The subtask trains the meta-parameters by performing meta-iteration using the gradient descent algorithm, thereby training the generative imitation reinforcement learning model, and generating the transportation trajectory based on the specific parameters of the task obtained from the training of the meta-parameters.

[0028] Furthermore, the process of acquiring road GPS point data and performing noise filtering to obtain GPS trajectory data specifically includes:

[0029] Collect road GPS point data, wherein the road GPS point data includes the coordinates of the road GPS points and the timestamp of the sampling time;

[0030] Noise points are removed from the road GPS point data to obtain useful road GPS point data;

[0031] The nearest neighbor matching algorithm is used to perform road matching on the useful road GPS point data to obtain GPS trajectory data.

[0032] In a second aspect, embodiments of the present invention provide a transportation trajectory generation device, comprising:

[0033] The acquisition module is used to acquire road GPS point data and perform noise filtering to obtain GPS trajectory data.

[0034] The modeling module is used to establish a state-action value function, a generation policy function, and a discrimination function based on the GPS trajectory data and the state set and action set in the five elements of the Markov decision process. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm. The parameters of the state-action value function, the generation policy function, and the discrimination function are updated to construct a generative imitation reinforcement learning model.

[0035] The generation module is used to employ a meta-learning algorithm, taking the state-action value function, generation policy function, and discrimination function as meta-parameters in the meta-learning algorithm. The meta-parameters are subjected to meta-iterative training, thereby training the generative imitation reinforcement learning model, and generating a transportation trajectory based on the specific parameters of the task obtained from the training of the meta-parameters.

[0036] Furthermore, the modeling module includes:

[0037] The first unit is used to acquire road traffic flow signals, and incorporate the traffic flow signals as shared external information into the GPS trajectory data to obtain new GPS trajectory data.

[0038] The second unit is used to establish a state-action value function, a generation strategy function, and a discrimination function based on the new GPS trajectory data and the state set and action set in the five elements of the Markov decision process. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm.

[0039] The third unit is used to parameterize the state-action value function, the generation policy function, and the discrimination function. The state-action value function and the generation policy function are updated through joint training, and their expressions are as follows:

[0040]

[0041] in, The loss function is the state-action value function; The loss function is the generation strategy function; c is the weight coefficient.

[0042] The parameter update process of the state-action value function conforms to the Q-learning algorithm as follows:

[0043]

[0044] in, Let E be the loss function of the state-action value function; E be the expected reward; Q be the loss function of the state-action value function. ψ (s t ,a t R(s) is the state-action value function at the current moment; t ,a t R(s) is the reward function. t ,a t )=-logD φ (s t ,a t ), D φ (s t ,a t) represents the discrimination function at the current moment; γ is the discount factor used to calculate the expected reward; π θ (a t+1 |s t+1 Q is the generation policy function for the next time step; ψ (s t+1 ,a t+1 ) is the state-action value function for the next time step; a t+1 This represents the action in the next moment;

[0045] The parameter update process of the generation policy function conforms to the gradient descent algorithm as follows:

[0046]

[0047] in, The loss function of the generation policy function; E is the expected reward; Q ψ (s t ,a t ) represents the state-action value function at the current moment; π θ (a t |s t H(π) is the generation strategy function at the current time; μ is a user-defined parameter; H(π) is the generation strategy function at the current time. θ ) represents the regularization term in the parameter update process of the generation strategy function;

[0048] The discrimination function is updated with parameters until it converges to the point where it is difficult to distinguish between the generated trajectory and the real trajectory, thereby constructing a generative imitation reinforcement learning model. The parameter update process of the discrimination function is as follows:

[0049]

[0050] in, E is the loss function of the discrimination function; π The expected reward for the generation strategy; The expected reward for the expert strategy; D(s) t ,a t ) is the discrimination function at the current moment.

[0051] Furthermore, the generation module includes:

[0052] A partitioning unit is used to divide the area where the transportation trajectory to be generated into multiple blocks, wherein the blocks are spatially adjacent and have similar road network structures; based on the task partitioning criteria of time and geographic similarity, the blocks and GPS trajectory data of the blocks at different time periods are defined as subtasks of the meta-learning algorithm;

[0053] The transformation unit is used to use the state-action value function, the generation strategy function, and the discrimination function as meta-parameters of the sub-task;

[0054] The training unit is used to train the meta-parameters by performing meta-iterations on the subtask using the gradient descent algorithm, thereby training the generative imitation reinforcement learning model, and generating the transportation trajectory based on the task-specific parameters obtained from the training of the meta-parameters.

[0055] Furthermore, the acquisition module includes:

[0056] The acquisition unit is used to acquire road GPS point data, wherein the road GPS point data includes the coordinates of the road GPS points and the timestamp of the sampling time.

[0057] The processing unit is used to remove noise points from the road GPS point data to obtain useful road GPS point data;

[0058] The matching unit is used to perform road matching on the useful road GPS point data using the nearest neighbor matching algorithm to obtain GPS trajectory data.

[0059] Thirdly, embodiments of the present invention provide a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the transportation trajectory generation method as described in any of the preceding claims.

[0060] Furthermore, embodiments of the present invention also provide a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the transportation trajectory generation method described above.

[0061] Compared with existing technologies, the present invention discloses a transportation trajectory generation method, apparatus, terminal, and storage medium. It acquires road GPS point data and performs noise filtering to obtain GPS trajectory data. Based on the GPS trajectory data and the state set and action set from the five elements of a Markov decision process, it establishes a state-action value function, a generation strategy function, and a discrimination function. A minimax game algorithm is used to use the discrimination function to identify the generated trajectory as a real trajectory. The parameters of the state-action value function, generation strategy function, and discrimination function are updated to construct a generative imitation reinforcement learning model. A meta-learning algorithm is employed, using the state-action value function, generation strategy function, and discrimination function as meta-parameters in the meta-learning algorithm. These meta-parameters undergo meta-iterative training to train the generative imitation reinforcement learning model. The transportation trajectory is generated based on the task-specific parameters obtained from the meta-parameter training. This method can construct a generative imitation reinforcement learning model, improve the model's accuracy, and generate urban transportation trajectories, thus adapting to complex travel environments. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating a transportation trajectory generation method provided in an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of the structure of a transportation trajectory generation device provided in an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] It should be noted that the terms "comprising" and "specific" in this invention, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0067] Please see Figure 1 , Figure 1 This is a flowchart illustrating a transportation trajectory generation method provided in an embodiment of the present invention. The transportation trajectory generation method includes steps S1 to S3:

[0068] S1: Obtain road GPS point data and perform noise filtering to obtain GPS trajectory data;

[0069] S2: Based on the GPS trajectory data and the state set and action set in the five elements of the Markov decision process, establish a state-action value function, a generation policy function and a discrimination function. Use the min-max game algorithm to use the discrimination function to identify the generated trajectory as a real trajectory, and update the parameters of the state-action value function, the generation policy function and the discrimination function to construct a generative imitation reinforcement learning model.

[0070] S3: Employ a meta-learning algorithm, using the state-action value function, generation policy function, and discrimination function as meta-parameters in the meta-learning algorithm. The meta-parameters undergo meta-iterative training to train the generative imitation reinforcement learning model, and the transportation trajectory is generated based on the specific parameters of the task obtained from the training of the meta-parameters.

[0071] In practice, road GPS point data is acquired and noise is filtered out to obtain GPS trajectory data. Vehicle GPS locations at different times on the road can be collected via cloud-based GPS as road GPS point data samples. The acquired road GPS point data is often presented in the form of time-series GPS point pairs, i.e., TJ:={(x1,y1,t1),…,(x n ,y n ,t n )|(x i ,y i ,t i )∈S}, where x i y i This represents the vehicle's GPS location coordinates, while t i The timestamp at the time of sampling indicates the GPS point pair data. GPS trajectory data is obtained by processing the GPS point pair data. The GPS trajectory data includes the GPS location coordinates and the timestamp at the time of sampling.

[0072] Based on the GPS trajectory data and the state and action sets from the five elements of a Markov decision process, a state-action value function, a generation policy function, and a discrimination function are established. A min-max game algorithm is used to apply the discrimination function to determine if the generated trajectory is a real trajectory. The parameters of the state-action value function, generation policy function, and discrimination function are updated to construct a generative imitation reinforcement learning model. The trajectory generation problem can be modeled as a Markov decision process. For example, the GPS trajectory data can be used as the state set of the Markov decision process, and the action sets of left turn, right turn, and straight ahead at road intersections can be used as the action set. The state-action value function, generation policy function, and discrimination function are then established. This embodiment of the invention uses a min-max game algorithm to find a balance between the generated trajectory and the trajectory's authenticity. The min-max game algorithm can be expressed as:

[0073]

[0074] Among them, the discriminant function D(s) t ,a t ) is used to identify state-action pairs, π is used to represent the generation policy, and E π and As the expected reward of the generative policy and the expert policy, the algorithm makes the generative policy π and the discriminant function D(s) equal. t ,a t While maximizing the expected reward of the policy, we hope to identify the function D(s). t ,a t The goal is to identify the generated trajectory as a real trajectory as much as possible. The parameters of the state-action value function, generation policy function, and discrimination function are updated to construct a generative imitation reinforcement learning model.

[0075] A meta-learning algorithm is adopted, in which the state-action value function, the generation policy function and the discrimination function are used as meta-parameters in the meta-learning algorithm. The meta-parameters are subjected to meta-iterative training, thereby training the generative imitation reinforcement learning model, and the transportation trajectory is generated based on the specific parameters of the task obtained from the training of the meta-parameters.

[0076] In the first preferred embodiment, step S2 involves establishing a state-action value function, a generation policy function, and a discrimination function based on the GPS trajectory data and the state set and action set from the five elements of the Markov decision process. The discrimination function is then used to identify the generated trajectory as a real trajectory using a minimax game algorithm. The parameters of the state-action value function, generation policy function, and discrimination function are updated to construct a generative imitation reinforcement learning model. Specifically, this includes:

[0077] The traffic flow signal of the road is acquired, and the traffic flow signal is incorporated into the GPS trajectory data as shared external information to obtain new GPS trajectory data;

[0078] Based on the new GPS trajectory data and the state set and action set in the five elements of the Markov decision process, a state-action value function, a generation strategy function, and a discrimination function are established. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm.

[0079] The state-action value function, the generation policy function, and the discrimination function are parameterized, and the state-action value function and the generation policy function are updated through joint training. Their expressions are as follows:

[0080]

[0081] in, The loss function is the state-action value function; The loss function is the generation strategy function; c is the weight coefficient.

[0082] The parameter update process of the state-action value function conforms to the Q-learning algorithm as follows:

[0083]

[0084] in, Let E be the loss function of the state-action value function; E be the expected reward; Q be the loss function of the state-action value function. ψ (s t ,a t R(s) is the state-action value function at the current moment; t ,a t R(s) is the reward function. t ,a t )=-logD φ (s t ,a t ), D φ (s t ,a t ) represents the discrimination function at the current moment; γ is the discount factor used to calculate the expected reward; π θ (a t+1 |s t+1 Q is the generation policy function for the next time step; ψ (s t+1 ,a t+1 ) is the state-action value function for the next time step; a t+1 This represents the action in the next moment;

[0085] The parameter update process of the generation policy function conforms to the gradient descent algorithm as follows:

[0086]

[0087] in, The loss function of the generation policy function; E is the expected reward; Q ψ (s t ,a t ) represents the state-action value function at the current moment; π θ (a t |s t H(π) is the generation strategy function at the current time; μ is a user-defined parameter; H(π) is the generation strategy function at the current time. θ ) represents the regularization term in the parameter update process of the generation strategy function;

[0088] The discrimination function is updated with parameters until it converges to the point where it is difficult to distinguish between the generated trajectory and the real trajectory, thereby constructing a generative imitation reinforcement learning model. The parameter update process of the discrimination function is as follows:

[0089]

[0090] in, E is the loss function of the discrimination function; π The expected reward for the generation strategy; The expected reward for the expert strategy; D(s) t ,a t ) is the discrimination function at the current moment.

[0091] It should be noted that traffic flow signals are calculated from the vehicle flow on each road and represent the congestion level on the road during the current time period. The new GPS trajectory data includes GPS location coordinates, the timestamp of the sampling time, and traffic flow signals, parameterized by the state-action value function Q. ψ (s t ,a t Generate policy function π θ (a t |s t ) and the discriminant function D φ (s t ,a t The imitation reinforcement learning task, s t a represents the state at the current moment. t This represents the action at the current moment. During the parameter update process of the state-action value function, the reward function is defined as R(s). t ,a t )=-logD φ (s t,a t Assigning a higher reward function to a generation strategy that is closer to the real trajectory distribution can ensure that the generation strategy learns the real trajectory distribution better, thereby achieving the effect of imitating expert data learning.

[0092] Considering the sparsity of expert policies, a canonical term is added during the parameter update process of the generated policy function to prevent the generated policy from directly copying the expert policy and falling into local optima. The loss functions for the state-action value function and the generated policy function are also considered. and joint training Parameters are updated in the form of the state-action value function Q. ψ (s t ,a t While continuously approximating the distribution of user travel trajectories, the generation strategy function generates transportation trajectories with the same distribution.

[0093] During the parameter update process of the discrimination function, the loss function of the discrimination function... This prompts the discriminant function D φ (s t ,a t The convergence to 0.5 indicates that the discriminant function has difficulty distinguishing between the generated trajectory and the real trajectory. This also means that the generation strategy can generate transportation trajectories with high accuracy, achieving the expected results.

[0094] In the second preferred embodiment, step S3 involves employing a meta-learning algorithm, using the state-action value function, generation policy function, and discrimination function as meta-parameters in the meta-learning algorithm. These meta-parameters undergo meta-iterative training to train the generative imitation reinforcement learning model. The transportation trajectory is generated based on the task-specific parameters obtained from the training of the meta-parameters. Specifically, this includes:

[0095] The area to be generated for transportation trajectories is divided into multiple blocks, and the blocks are spatially adjacent and have similar road network structures to ensure the similarity of the traffic flow signals. Based on the task division criteria of time and geographic similarity, the blocks and GPS trajectory data of the blocks at different time periods are defined as subtasks of the meta-learning algorithm.

[0096] The state-action value function, the generation policy function, and the discrimination function are used as meta-parameters of the sub-task;

[0097] The subtask trains the meta-parameters by performing meta-iteration using the gradient descent algorithm, thereby training the generative imitation reinforcement learning model, and generating the transportation trajectory based on the specific parameters of the task obtained from the training of the meta-parameters.

[0098] It should be noted that in meta-learning, subtask partitioning has a crucial impact on the model's generalization ability. This embodiment uses a task partitioning criterion based on temporal and geographical similarity to train an optimal model across multiple tasks.

[0099] In practical implementation, based on the observation of the temporal heterogeneity of traffic flow in the same area, urban transportation trajectories are divided according to temporal and spatial correlation, and the city is divided into N different urban blocks R = {r1, r2, ..., r...} according to geographical location. N}, extracting M different times D = {d1, d2, ..., d} from each city block N The corresponding urban GPS trajectory data is used. The sub-task set is defined as T = R × D, representing the task of generating urban GPS trajectories in a specific time and space under this trajectory division. For urban block division, spatially adjacent urban areas and similar road network structures are selected to ensure traffic flow similarity. For time division, different date attributes, including holidays and weekdays, are selected as time nodes, and the 24 hours of a day are further divided into daytime, nighttime, and early morning to obtain the heterogeneity of urban traffic flow at different times. In general, under each sub-task, the urban GPS trajectories are different but also have interdependent relationships in travel strategies. For example, despite the heterogeneity of trajectories under different sub-tasks, users tend to choose urban roads without congestion. To increase the model's generalization ability on sparse data, a meta-learning algorithm is considered. The state-action value function, generation policy function, and discrimination function are used as meta-parameters of the sub-task. Through a meta-iterative training process on similar tasks, the gradient descent algorithm is used to train the meta-parameters, thereby training the generative imitation reinforcement learning model. The specific parameters of the task trained based on the meta-parameters are used to generate the transportation trajectory. Meta-learning parameter θ j Q is the state-action value function at the j-th iteration. ψ (s t ,a t ), Generate policy function π θ (a t |s t ) and the discriminant function D φ (s t ,a t ), that is, θ j := {ψ,θ,φ}. The meta-iterative process can be described as:

[0100] θ j+1 =θ j -β(θ i -θ j ),

[0101] Where, θ j+1Represents the meta-parameters at the (j+1)th iteration; β is the iteration step size; θ i These are the specific parameters for the task. For a given set of subtasks T = R × D generated from urban GPS trajectories at different times and spaces, consider a specific subtask T. i The gradient descent algorithm is used to iterate over subtask T. i Task specific parameters Within each individual subtask, progress is made by exploring the task in small steps. This provides some valid information and guides the meta-learning of parameters θ. j The updates are used to generate high-quality urban transportation tracks.

[0102] In the third preferred embodiment, step S1, which involves acquiring road GPS point data and performing noise filtering to obtain GPS trajectory data, specifically includes:

[0103] Collect road GPS point data, wherein the road GPS point data includes the coordinates of the road GPS points and the timestamp of the sampling time;

[0104] Noise points are removed from the road GPS point data to obtain useful road GPS point data;

[0105] The nearest neighbor matching algorithm is used to perform road matching on the useful road GPS point data to obtain GPS trajectory data.

[0106] In practical implementation, road GPS point data is collected, which includes the coordinates of the road GPS points and the timestamp at the time of sampling. Urban transportation trajectories are often represented in the form of time-series GPS point pairs, i.e., TJ:={(x1,y1,t1),…,(x n ,y n ,t n )|(x i ,y i ,t i )∈S}, where x i y i The GPS sequence coordinates representing the city's trajectory, and t iThe timestamp indicates the sampling time. This type of trajectory has significant limitations in representing the spatial connectivity of locations; for example, it's difficult to determine whether there are road connections between adjacent GPS points. More often, the trajectory data contains intolerable GPS measurement noise, such as GPS data falling outside urban roads or discrete data spanning different roads. To eliminate noise between GPS representations, noisy points in the road GPS point data are removed to obtain useful road GPS point data, transforming the trajectory from discrete GPS points to continuous roads. Nearest neighbor matching is used for road matching, while considering the connectivity limitations of the road trajectory, filtering out unnecessary noise points to obtain the final GPS trajectory data.

[0107] Please see Figure 2 , Figure 2 This is a schematic diagram of a transportation trajectory generation device provided in an embodiment of the present invention. The transportation trajectory generation device includes:

[0108] The acquisition module 21 is used to acquire road GPS point data and perform noise filtering to obtain GPS trajectory data;

[0109] Modeling module 22 is used to establish a state-action value function, a generation policy function, and a discrimination function based on the GPS trajectory data and the state set and action set in the five elements of the Markov decision process. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm. The parameters of the state-action value function, the generation policy function, and the discrimination function are updated to construct a generative imitation reinforcement learning model.

[0110] The generation module 23 is used to employ a meta-learning algorithm, taking the state-action value function, generation policy function, and discrimination function as meta-parameters in the meta-learning algorithm. The meta-parameters are subjected to meta-iterative training, thereby training the generative imitation reinforcement learning model, and generating a transportation trajectory based on the specific parameters of the task obtained from the training of the meta-parameters.

[0111] In the first embodiment, the modeling module 22 includes:

[0112] The first unit is used to acquire road traffic flow signals, and incorporate the traffic flow signals as shared external information into the GPS trajectory data to obtain new GPS trajectory data.

[0113] The second unit is used to establish a state-action value function, a generation strategy function, and a discrimination function based on the new GPS trajectory data and the state set and action set in the five elements of the Markov decision process. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm.

[0114] The third unit is used to parameterize the state-action value function, the generation policy function, and the discrimination function. The state-action value function and the generation policy function are updated through joint training, and their expressions are as follows:

[0115]

[0116] in, The loss function is the state-action value function; The loss function is the generation strategy function; c is the weight coefficient.

[0117] The parameter update process of the state-action value function conforms to the Q-learning algorithm as follows:

[0118]

[0119] in, Let E be the loss function of the state-action value function; E be the expected reward; Q be the loss function of the state-action value function. ψ (s t ,a t R(s) is the state-action value function at the current moment; t ,a t R(s) is the reward function. t ,a t )=-log D φ (s t ,a t ), D φ (s t ,a t ) represents the discrimination function at the current moment; γ is the discount factor used to calculate the expected reward; π θ (a t+1 |s t+1 Q is the generation policy function for the next time step; ψ (s t+1 ,a t+1 ) is the state-action value function for the next time step; a t+1 This represents the action in the next moment;

[0120] The parameter update process of the generation policy function conforms to the gradient descent algorithm as follows:

[0121]

[0122] in, The loss function of the generation policy function; E is the expected reward; Q ψ (s t ,a t ) represents the state-action value function at the current moment; πθ (a t |s t H(π) is the generation strategy function at the current time; μ is a user-defined parameter; H(π) is the generation strategy function at the current time. θ ) represents the regularization term in the parameter update process of the generation strategy function;

[0123] The discrimination function is updated with parameters until it converges to the point where it is difficult to distinguish between the generated trajectory and the real trajectory, thereby constructing a generative imitation reinforcement learning model. The parameter update process of the discrimination function is as follows:

[0124]

[0125] in, E is the loss function of the discrimination function; π The expected reward for the generation strategy; The expected reward for the expert strategy; D(s) t ,a t ) is the discrimination function at the current moment.

[0126] In the second embodiment, the generation module 23 includes:

[0127] A partitioning unit is used to divide the area where the transportation trajectory to be generated into multiple blocks, wherein the blocks are spatially adjacent and have similar road network structures; based on the task partitioning criteria of time and geographic similarity, the blocks and GPS trajectory data of the blocks at different time periods are defined as subtasks of the meta-learning algorithm;

[0128] The transformation unit is used to use the state-action value function, the generation strategy function, and the discrimination function as meta-parameters of the sub-task;

[0129] The training unit is used to train the meta-parameters by performing meta-iterations on the subtask using the gradient descent algorithm, thereby training the generative imitation reinforcement learning model, and generating the transportation trajectory based on the task-specific parameters obtained from the training of the meta-parameters.

[0130] In the third embodiment, the acquisition module 21 includes:

[0131] The acquisition unit is used to acquire road GPS point data, wherein the road GPS point data includes the coordinates of the road GPS points and the timestamp of the sampling time.

[0132] The processing unit is used to remove noise points from the road GPS point data to obtain useful road GPS point data;

[0133] The matching unit is used to perform road matching on the useful road GPS point data using the nearest neighbor matching algorithm to obtain GPS trajectory data.

[0134] The transportation trajectory generation device provided in this embodiment of the invention can realize all the processes of the transportation trajectory generation method of any of the above embodiments. The functions and technical effects of each module and unit in the device are the same as the functions and technical effects of the transportation trajectory generation method of the above embodiments, and will not be repeated here.

[0135] See Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. The terminal device 3 of this embodiment includes: a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the computer program, it implements the steps in the above-described transportation trajectory generation method embodiment. Alternatively, when the processor 31 executes the computer program, it implements the functions of each module in the above-described transportation trajectory generation device embodiment.

[0136] For example, the computer program may be divided into one or more modules, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device 3.

[0137] The terminal device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The terminal device 3 may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components. For example, the terminal device 3 may also include input / output devices, network access devices, buses, etc.

[0138] The processor 31 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor 31 is the control center of the terminal device 3, connecting all parts of the terminal device 3 via various interfaces and lines.

[0139] The memory 32 can be used to store the computer programs and / or modules. The processor 31 implements various functions of the terminal device 3 by running or executing the computer programs and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0140] If the modules integrated in the terminal device 3 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 31, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0141] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0142] This invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when running, controls the device containing the computer-readable storage medium to execute the transportation trajectory generation method as described above.

[0143] In summary, this invention discloses a method, apparatus, device, and storage medium for generating transportation trajectories. It obtains GPS trajectory data by acquiring road GPS point data and performing noise filtering. Based on the GPS trajectory data and the state set and action set from the five elements of a Markov decision process, a state-action value function, a generation strategy function, and a discrimination function are established. A minimax game algorithm is used to use the discrimination function to identify the generated trajectory as a real trajectory. The parameters of the state-action value function, generation strategy function, and discrimination function are updated to construct a generative imitation reinforcement learning model. A meta-learning algorithm is employed, using the state-action value function, generation strategy function, and discrimination function as meta-parameters in the meta-learning algorithm. These meta-parameters undergo meta-iterative training to train the generative imitation reinforcement learning model. The transportation trajectory is generated based on the task-specific parameters obtained from the meta-parameter training. This method can construct a generative imitation reinforcement learning model, improve the model's accuracy, and generate urban transportation trajectories, thus adapting to complex travel environments.

[0144] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating a transportation trajectory, characterized in that, Includes the following steps: Obtain road GPS point data and perform noise filtering to obtain GPS trajectory data; Based on the GPS trajectory data and the state set and action set in the five elements of the Markov decision process, a state-action value function, a generation policy function, and a discrimination function are established. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm. The parameters of the state-action value function, the generation policy function, and the discrimination function are updated to construct a generative imitation reinforcement learning model. A meta-learning algorithm is adopted, in which the state-action value function, the generation policy function and the discrimination function are used as meta-parameters in the meta-learning algorithm. The meta-parameters are subjected to meta-iterative training, thereby training the generative imitation reinforcement learning model, and the transportation trajectory is generated based on the specific parameters of the task obtained from the training of the meta-parameters. Specifically, based on the GPS trajectory data and the state set and action set from the five elements of the Markov decision process, a state-action value function, a generation policy function, and a discrimination function are established. A minimax game algorithm is used to apply the discrimination function to identify the generated trajectory as a real trajectory. The parameters of the state-action value function, generation policy function, and discrimination function are updated to construct a generative imitation reinforcement learning model. This specifically includes: The traffic flow signal of the road is acquired, and the traffic flow signal is incorporated into the GPS trajectory data as shared external information to obtain new GPS trajectory data; Based on the new GPS trajectory data and the state set and action set in the five elements of the Markov decision process, a state-action value function, a generation strategy function, and a discrimination function are established. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm. The state-action value function, the generation policy function, and the discrimination function are parameterized, and the state-action value function and the generation policy function are updated through joint training. Their expressions are as follows: , ， in, The loss function is the state-action value function; The loss function is the generation strategy function; c is the weight coefficient. The parameter update process of the state-action value function conforms to the Q-learning algorithm as follows: ， in, The loss function is the state-action value function; For expected reward; The state-action value function at the current moment; For the reward function, , The discrimination function at the current moment; As a discount factor when calculating the expected reward; The generation strategy function for the next time step; The state-action value function for the next moment; This represents the action in the next moment; The parameter update process of the generation policy function conforms to the gradient descent algorithm as follows: ， in, The loss function for the generation policy function; The expected reward; The state-action value function at the current moment; The generation strategy function at the current moment; For custom parameters; The regularization term for the parameter update process of the generation strategy function; The discrimination function is updated with parameters until it converges to the point where it is difficult to distinguish between the generated trajectory and the real trajectory, thereby constructing a generative imitation reinforcement learning model. The parameter update process of the discrimination function is as follows: ， in, The loss function is the discrimination function. The expected reward for the generation strategy; Expected reward for expert strategies; Let be the discrimination function at the current moment.

2. The transportation trajectory generation method as described in claim 1, characterized in that, The meta-learning algorithm is employed, using the state-action value function, generation policy function, and discrimination function as meta-parameters. These meta-parameters undergo meta-iterative training to train the imitation reinforcement learning model. The transportation trajectory is generated based on the task-specific parameters obtained from the meta-parameter training. Specifically, this includes: The area to be generated for transportation trajectories is divided into multiple blocks, which are spatially adjacent and have similar road network structures. Based on the task division criteria of time and geographic similarity, the GPS trajectory data of the blocks and the blocks at different time periods are defined as subtasks of the meta-learning algorithm. The state-action value function, the generation policy function, and the discrimination function are used as meta-parameters of the sub-task; The subtask trains the meta-parameters by performing meta-iteration using the gradient descent algorithm, thereby training the generative imitation reinforcement learning model, and generating the transportation trajectory based on the specific parameters of the task obtained from the training of the meta-parameters.

3. The transportation trajectory generation method as described in claim 1, characterized in that, The process of acquiring road GPS point data and performing noise filtering to obtain GPS trajectory data specifically includes: Collect road GPS point data, wherein the road GPS point data includes the coordinates of the road GPS points and the timestamp of the sampling time; Noise points are removed from the road GPS point data to obtain useful road GPS point data; The nearest neighbor matching algorithm is used to perform road matching on the useful road GPS point data to obtain GPS trajectory data.

4. A transportation trajectory generation device, characterized in that, include: The acquisition module is used to acquire road GPS point data and perform noise filtering to obtain GPS trajectory data. The modeling module is used to establish a state-action value function, a generation policy function, and a discrimination function based on the GPS trajectory data and the state set and action set in the five elements of the Markov decision process. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm. The parameters of the state-action value function, the generation policy function, and the discrimination function are updated to construct a generative imitation reinforcement learning model. The generation module is used to employ a meta-learning algorithm, taking the state-action value function, generation policy function, and discrimination function as meta-parameters in the meta-learning algorithm. The meta-parameters are subjected to meta-iterative training, thereby training the generative imitation reinforcement learning model, and generating a transportation trajectory based on the specific parameters of the task obtained from the training of the meta-parameters. The modeling module includes: The first unit is used to acquire road traffic flow signals, and incorporate the traffic flow signals as shared external information into the GPS trajectory data to obtain new GPS trajectory data. The second unit is used to establish a state-action value function, a generation strategy function, and a discrimination function based on the new GPS trajectory data and the state set and action set in the five elements of the Markov decision process. The discrimination function is used to identify the generated trajectory as the real trajectory using the minimax game algorithm. The third unit is used to parameterize the state-action value function, the generation policy function, and the discrimination function. The state-action value function and the generation policy function are updated through joint training, and their expressions are as follows: , ， in, The loss function is the state-action value function; The loss function is the generation strategy function; c is the weight coefficient. The parameter update process of the state-action value function conforms to the Q-learning algorithm as follows: ， in, The loss function is the state-action value function; For expected reward; The state-action value function at the current moment; For the reward function, , The discrimination function at the current moment; As a discount factor when calculating the expected reward; The generation strategy function for the next time step; The state-action value function for the next moment; This represents the action in the next moment; The parameter update process of the generation policy function conforms to the gradient descent algorithm as follows: ， in, The loss function for the generation policy function; The expected reward; The state-action value function at the current moment; The generation strategy function at the current moment; For custom parameters; The regularization term for the parameter update process of the generation strategy function; The discrimination function is updated with parameters until it converges to the point where it is difficult to distinguish between the generated trajectory and the real trajectory, thereby constructing a generative imitation reinforcement learning model. The parameter update process of the discrimination function is as follows: ， in, The loss function is the discrimination function. The expected reward for the generation strategy; Expected reward for expert strategies; Let be the discrimination function at the current moment.

5. The transportation trajectory generation device as described in claim 4, characterized in that, The generation module includes: A partitioning unit is used to divide the area where the transportation trajectory to be generated into multiple blocks, wherein the blocks are spatially adjacent and have similar road network structures; based on the task partitioning criteria of time and geographic similarity, the blocks and GPS trajectory data of the blocks at different time periods are defined as subtasks of the meta-learning algorithm; The transformation unit is used to use the state-action value function, the generation strategy function, and the discrimination function as meta-parameters of the sub-task; The training unit is used to train the meta-parameters by performing meta-iterations on the subtask using the gradient descent algorithm, thereby training the generative imitation reinforcement learning model, and generating the transportation trajectory based on the task-specific parameters obtained from the training of the meta-parameters.

6. The transportation trajectory generation device as described in claim 4, characterized in that, The acquisition module includes: The acquisition unit is used to acquire road GPS point data, wherein the road GPS point data includes the coordinates of the road GPS points and the timestamp of the sampling time. The processing unit is used to remove noise points from the road GPS point data to obtain useful road GPS point data; The matching unit is used to perform road matching on the useful road GPS point data using the nearest neighbor matching algorithm to obtain GPS trajectory data.

7. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the transportation trajectory generation method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the transportation trajectory generation method as described in any one of claims 1 to 3.

Images