A method for adaptively adjusting a map matching search radius based on reinforcement learning

Abstract

The application discloses a method for adaptively adjusting a map matching search radius based on reinforcement learning. The Q learning algorithm in the reinforcement learning is used to complete adaptive adjustment of the search radius, agent, environment, state and action variables of the Q learning algorithm for adaptively adjusting the search radius are determined, a double-target reward function considering the size of a search candidate road section set and the map matching accuracy performance is constructed, and an action value function based on a strategy is formulated to determine an optimal search radius strategy. In addition, a heuristic algorithm is used to screen GPS samples to select a certain amount of training data, so as to improve the training effect of the reinforcement learning. The application can adaptively determine the search radius according to the position information of the GPS, so as to improve the calculation efficiency of the map matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology and data processing technology, in particular to a search candidate road segment technology in map matching, which can be applied to positioning, navigation, route planning and other location-based services, and more particularly to a method for adaptively adjusting a search radius in map matching based on reinforcement learning. BACKGROUND

[0002] With the increasing popularity and application of positioning technologies and devices such as GPS sensors (such as navigation and route planning), a large amount of GPS trajectory data is collected and applied in different fields. Due to the positioning error and sampling error of the readings of the GPS sensor, the original GPS trajectory needs to be preprocessed before use. Map matching is one of the most important preprocessing tasks, and its purpose is to determine the actual driving route of a given GPS trajectory through the matching of the GPS positioning sequence and the underlying road network. The basic idea of the map matching algorithm is to determine the actual driving road of the vehicle according to the GPS position of the vehicle and the corresponding road network data information, map a series of ordered original GPS sampling points of the vehicle to the road network, and find the true position of the original GPS sampling points in the road network through the map matching algorithm. The main process of the map matching algorithm is as follows (see Figure 1 ):

[0003] (1) determining the candidate road segments of the sampling points based on the original GPS sampling points and a given search radius;

[0004] (2) feature extraction of the candidate road segments and modeling of the spatio-temporal relationship between the candidate road segments;

[0005] (3) selecting the optimal candidate road segment sequence as the matching path.

[0006] A number of map matching methods have been proposed, most of which are based on Hidden Markov Model (HMM). HMM is good at modeling GPS position sequences by incorporating additional features such as road connectivity and travel direction. The paper (Paul Newson and John Krumm. 2009. Hidden Markov map matching through noise and sparseness. In Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (GIS'09). Association for Computing Machinery, New York, NY, USA, 336-343.) links the position information of raw GPS samples and the road network information of open source map OpenStreetMap (OSM), calculates the probability value of raw GPS samples to the nearby road segment (i.e. observation probability), and the transition probability between candidate road segments, and finally selects the candidate road segment sequence with the maximum cumulative probability value as the final matching path. The HMM-based map matching algorithm is sensitive to the position information of raw GPS samples and the reasonableness of the connectivity between road segments. The main process of HMM map matching is as follows:

[0007] (1) Index the road network using R-tree or quadtree index, and search the candidate road segments of each GPS sample point using the index;

[0008] (2) Calculate the observation probability and transition probability of the candidate road segment state;

[0009] (3) Calculate the optimal matching path using Viterbi algorithm.

[0010] It is observed that the key to reducing the HMM map matching computation overhead is to reduce the number of candidate road segments for each GPS sample point. However, existing work uses a fixed search radius parameter to search for candidate road segments for all GPS sample points (Zhihao Shen, Wan Du, Xi Zhao, and Jianhua Zou. 2020. DMM: fast map matching for cellular data. Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. Association for Computing Machinery, New York, NY, USA, Article 60, 1-14.). In practice, we observe that GPS positioning errors vary greatly in different cities. GPS positioning has larger errors in some urban canyon areas, but performs better in other urban areas. In the observation data collected in the GPS positioning accurate area, using a fixed search radius will introduce irrelevant road segments and increase the computation overhead; while in the area with larger errors, a too small search radius may not find the correct road segment, reducing the accuracy of map matching. SUMMARY

[0011] The present application aims to solve the above-mentioned problems in the prior art, and provides a method for adaptively adjusting the search radius of map matching based on reinforcement learning.

[0012] To achieve the above technical purposes, the present application adopts the following technical solutions,

[0013] A method for adaptively adjusting the search radius of map matching based on reinforcement learning, comprising:

[0014] Divide the urban road network based on a hexagonal grid of a preset size;

[0015] Take the road network spatial index as the environment and the map matching algorithm as the agent, for a given GPS sample in each hexagonal grid, take different search radius values as the state, take search radius expansion, invariance and reduction as the action, and take the principle of containing the correct road segment and the candidate road segment set as small as possible to establish a reward function, and model the adaptive adjustment of the search radius of each hexagonal grid as a Markov decision process;

[0016] Obtain the optimal search radius of all hexagonal grids using the Q-learning algorithm.

[0017] As a preferred implementation, the states with different search radius values include: after setting the maximum search radius value, discretize the search radius value with a preset step size to obtain a plurality of states represented based on the search radius value.

[0018] As a preferred implementation, the three actions of search radius expansion, invariability and reduction are correspondingly assigned values to represent the corresponding actions by numerical values.

[0019] As a preferred implementation, the reward function is defined as follows:

[0020]

[0021] wherein, is the current state s of the GPS sample g i is the corresponding candidate road segment set; G g is the correct candidate road segment of the GPS sample g, obtained by map matching; or 0, indicating whether the road segment G is contained g .

[0022] As a preferred implementation, the way to obtain the optimal search radius of all hexagonal grids by using the Q-learning algorithm is as follows:

[0023] Taking the GPS samples contained in each hexagonal grid as input, the state is initialized as the maximum search radius, and then the state-action value function is updated based on the immediate reward function value for each GPS sample. After all GPS samples are traversed, the state-action table is output.

[0024] Preferably, in the learning process, an action is selected by an ε-greedy strategy to obtain a new state.

[0025] Preferably, for the case of missing the correct candidate road segment, the search radius is gradually expanded by a preset step size to search for the candidate road segment.

[0026] As a preferred implementation, the GPS samples are screened based on the following method:

[0027] A preset threshold of the number of screening trajectories is set, and the trajectory screening operation is performed within the threshold range:

[0028] Traverse the trajectory data, and for each trajectory, establish a sequence of hexagonal grids containing GPS samples on the trajectory;

[0029] For each hexagon grid, the trajectory selection is performed in the order of the number of trajectories contained therein from less to more. After a trajectory is selected for a hexagon grid, the hexagon grid through which the trajectory passes is cleared, and the trajectory selection is continued in the order of the number of trajectories contained in the remaining grids from less to more until all the hexagon grids are covered by the selected trajectories.

[0030] As a preferred embodiment, when a hexagon grid contains two or more trajectories, the trajectory containing the most GPS samples is selected.

[0031] As a preferred embodiment, if the number of trajectories does not reach the threshold value after the screening, the trajectory selection is repeated until the number of trajectories reaches the threshold value.

[0032] The present application proposes a method for adaptively adjusting the search radius of map matching based on reinforcement learning. First, the problem of adaptively adjusting the search radius is modeled as a Markov decision process, and reinforcement learning is used to solve the problem. Then, the agent, environment, state, action variable and reward function in reinforcement learning are determined. Next, the training framework of Q-learning in reinforcement learning is determined. Finally, a heuristic algorithm is proposed to select training data to improve the training effect of reinforcement learning. The method of the present application can adaptively adjust the search radius according to the GPS positioning error in different regions, and improve the calculation efficiency and matching accuracy of map matching. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The map matching algorithm framework is shown in FIG. 1.

[0034] Figure 2 The adaptive adjustment of the search radius based on reinforcement learning is shown in FIG. 2.

[0035] Figure 3 The hexagon hierarchical structure of road network data is shown in FIG. 3.

[0036] Figure 4 The flowchart of the adaptive adjustment of the search radius based on reinforcement learning is shown in FIG. 4.

[0037] Figure 5 The flowchart of the trajectory selection method for covering road network based on the heuristic algorithm is shown in FIG. 5. DETAILED DESCRIPTION

[0038] The method of the present application is further described below in combination with the drawings and specific embodiments. The flowchart of the method for adaptively adjusting the search radius of map matching is as follows:

[0039] 1. Markov decision process modeling (see FIG. 6) Figure 2 )

[0040] First, the city road network is divided into hexagons of a certain size (see FIG. 7) Figure 3Due to differences in GPS positioning performance, each hexagonal region will have an optimal search radius. Specifically, different search radius values ​​γ are used... i Perform a road network spatial index query on a given GPS sample within each hexagon, and check the size and accuracy of the candidate road segment set. i The optimal settings minimize the candidate road segment set for most GPS samples and include the "correct" candidate road segments, achieving a minimum for each hexagonal H. i Both have the optimal search radius γ i The purpose is to achieve this. Therefore, we model the adaptive search radius adjustment problem as a Markov decision process, with the following modeling parameters:

[0041] (1) The road network spatial index is considered as the environment, and the map matching algorithm is considered as the agent. The specific definitions of the state, action, and reward function for the optimal search radius determination problem for each hexagonal region are as follows:

[0042] 1) State s i Each possible search radius value is treated as a state. Considering the large GPS error offset in urban areas, we set a maximum search radius value maxγ. r The distance is 100 meters, and it is discretized with an interval Δ of 5 meters. Therefore, we have a total of 20 states, namely S = [5, 10, 15, ..., 95, 100].

[0043] 2) Action a i Given the current state s i The agent can have three possible actions to adjust the search radius γ. i These represent increasing the search radius, keeping the search radius constant, and decreasing the search radius, respectively. Where, a i =1 means that s i Increase Δ; a i =-1 means make s i Reduce Δ; a i =0 then keep s i Unchanged. Note state s. i =5 has only two actions, namely a i =1 or 0, state s i =100 also only has two actions, namely a i = -1 or 0.

[0044] 3) Reward r i As feedback from the environment, the reward r i This is used to evaluate the effect of a given action in the current state. In this problem setting, we hope to retrieve a candidate road segment set C with the smallest possible capacity and containing accurate road segments from the road network spatial index of any GPS sample g. That is, the design of the reward function must satisfy two requirements:

[0045] a. state s i The accurate road segment must be searched to ensure the accuracy of map matching;

[0046] b. state s i The candidate road segment set searched is as small as possible, so that the time complexity of map matching dynamic programming is minimized. In particular, the candidate road segment set C must contain the "correct" candidate road segment G of g g .

[0047] Therefore, the reward function is defined as follows:

[0048]

[0049] wherein, represents whether it contains the "correct" road segment G g . If it contains, otherwise refers to the number of candidate road segments of the GPS sample g under the condition that the search radius is s i . The "correct" road segment G of the GPS sample g g can be obtained by map matching.

[0050] (2) The goal of the Markov decision process is to find an optimal strategy based on the estimation of the agent state-action value function (i.e. Q table) to guide the decision-making of each state. Based on the above Markov decision process modeling, the reinforcement learning Q learning algorithm is used to learn the best search radius of all hexagonal grids. In the experimental process, first, according to the position of the GPS sample, different GPS samples are divided into different hexagons, and then the GPS samples (denoted as i ) of each hexagon H i are used to learn the Q table of each hexagon H i .

[0051] 2. Adaptive search radius Q learning framework

[0052] Figure 4 The Q learning framework for determining the adaptive search radius of the hexagon H i is introduced (wherein N is the total number of hexagons H i , and t is the total number of GPS samples contained by H i ). The algorithm takes the GPS sample as input and outputs the Q table of the hexagon H max after reinforcement Q learning. The Q table is initialized to zero, and then the Q table is continuously learned according to the training data. The state s is initialized to the maximum search radius, i.e. S i, retrieve each GPS sample g and randomly select an action a with an ε-greedy policy i Thus, a new state s is obtained i And interact with the road network spatial index to calculate the immediate reward. The total Q value is updated as:

[0053]

[0054] where α is the learning rate and β is the discount factor.

[0055] After learning the Q table of hexagon H i , we can obtain the optimal search radius γ i of the GPS sample in hexagon H i . i γ i is the state with the highest Q value, that is:

[0056]

[0057] In principle, such a search radius γ i can return a refined candidate set for most GPS samples. And for the special case of missing the "correct" candidate road segment, gradually expand the search radius γ i = γ i + Δ to search for the next candidate road segment with a step size of Δ = 5 meters.

[0058] 3. Heuristic training data selection algorithm

[0059] Reinforcement learning requires trajectory data that has been map-matched as training samples to learn the best search radius for each hexagon. However, map matching itself is time-consuming, so selecting a batch of suitable trajectory data for fast map matching to provide enough training data for reinforcement learning is a problem to be solved.

[0060] If random techniques are simply used to select training trajectory data, some hexagons may not have enough GPS sample data for training, making it impossible to learn the search radius. In addition, if some hexagon regions only contain a few GPS samples with errors, the learned search radius will not have good generalization. Therefore, how to select the training data for reinforcement learning to ensure the quantity and quality of the training data and make them cover the entire road network as comprehensively as possible.

[0061] Since the reinforcement Q learning model needs map-matched GPS samples to calculate the Q table of all hexagons, we need to select a GPS trajectory set that covers the road network as much as possible for map matching. There are two requirements for the selection of the original trajectory:

[0062] (1) The computation cost of map matching is high. It is desirable to use as few trajectory data as possible for reinforcement Q-learning.

[0063] (2) The GPS samples of selected trajectories should cover all the hexagons in the road network, so that each hexagon can learn the Q-table.

[0064] Therefore, we propose a heuristic trajectory data selection method that greedily selects original trajectories to cover all the hexagons (see Figure 5 ). The specific steps of the training data selection algorithm are as follows:

[0065] Step 1: Given the expected number of original trajectories B to be selected, i.e., the size of the trajectory dataset for reinforcement learning is specified.

[0066] Step 2: Traverse the trajectory data, and for each trajectory T j in the candidate set T j , convert the sequence of GPS samples of each trajectory T j to a sequence of hexagon grids based on the location information, denoted as T j .lh.

[0067] Step 3: First, record the trajectory number H i .lt that passes through the hexagon grid h i , and the number of trajectories len(H i .lt) that pass through the hexagon; then use a list H list to record the hexagon number H i and the trajectory number H i .lt that passes through the hexagon; finally, sort the list H list in ascending order according to len(H i .lt).

[0068] Step 4: According to the order of the list H list , select trajectories in H i .lt one by one, and select one trajectory for each hexagon to achieve local optimization. If a hexagon contains two or more trajectories, select the one with the most GPS samples. Specifically, first find the hexagon H i with the fewest trajectories passing through it, then select a trajectory T j from H i .lt, and then remove the hexagon grids that the trajectory T j passes through. Continue selecting trajectories from the remaining hexagon grids in order of the number of trajectories they contain, from fewest to most, according to the list H list . After traversing the list H list , the number of selected trajectories is len(H i ), which ensures that the GPS trajectories cover all the hexagons H.

[0069] Fifth step: if the number of trajectories does not reach the threshold B after screening, repeat the fourth step until the number of trajectories reaches the threshold.

[0070] According to the experimental results, the coverage rate of the road network is only 70% for 2000 randomly selected trajectories, while the coverage rate has reached 100% for only 1000 trajectories selected by the heuristic method. Although the heuristic algorithm only selects 2% of the trajectories, the entire road network has been completely covered, which not only shortens the training time, but also improves the training quality.

[0071] Appendix: Symbolic annotation involved in the method:

[0072]

[0073]

Claims

1. A method for adaptively adjusting a map-matching search radius based on reinforcement learning, characterized in that, include: The urban road network is divided based on a hexagonal grid of preset size; Using the road network spatial index as the environment and the map matching algorithm as the agent, for each GPS sample within a hexagonal grid, different search radius values ​​are used as states, and expanding, keeping the search radius constant, and shrinking are used as actions. A reward function is established based on the principle of including the correct road segments and minimizing the set of candidate road segments. The problem of adaptively adjusting the search radius of each hexagonal grid is modeled as a Markov decision process. The optimal search radius for all hexagonal grids is obtained using the Q-learning algorithm.

2. The method of claim 1, wherein, The use of different search radius values ​​as states includes: after setting a maximum search radius value, discretizing the search radius value with a preset step size to obtain several states represented by the search radius value.

3. The method of claim 1, wherein, The search radius is expanded, kept constant, and reduced, and corresponding values ​​are assigned to represent the corresponding actions.

4. The method of claim 1, wherein, The reward function is defined as follows: wherein, is the current state s of GPS sample g i corresponding candidate segment set; G g is the correct candidate segment of GPS sample g, obtained by map matching; or 0, indicating whether the segment G is contained g .

5. The method of claim 1, wherein, The method for obtaining the optimal search radius for all hexagonal grids using the Q-learning algorithm is as follows: Using the GPS samples contained in each hexagonal grid as input, the state is first initialized to the maximum search radius. Then, for each GPS sample, the state-action value function is updated sequentially based on the instantaneous reward function value. After traversing all GPS samples, the state-action table is output.

6. The method of claim 5, wherein, During the learning process, an action is selected using the ε-greedy strategy to obtain a new state.

7. The method of claim 5, wherein, If the correct candidate road segment is missed, the search radius is gradually expanded by a preset step size to search for candidate road segments.

8. The method of claim 1, wherein, The GPS samples were selected based on the following method: Set a preset threshold for the number of tracks to filter, and perform track filtering operations within the threshold range: Traverse the trajectory data and create a sequence of hexagonal grids containing GPS samples on each trajectory; For each hexagonal grid, trajectory selection is performed in ascending order of the number of trajectories it contains. After selecting a trajectory that passes through a hexagonal grid, the hexagonal grids that the trajectory passed through are cleared. Trajectory selection continues from the remaining grids in ascending order of the number of trajectories they contain, until the selected trajectory covers all hexagonal grids.

9. The method of claim 8, wherein, When a hexagonal grid contains two or more trajectories, select the trajectory that contains the most GPS samples.

10. The method of claim 8, wherein, If the number of trajectories does not reach the threshold after filtering, repeat the trajectory selection until the number of trajectories reaches the threshold.

Images