Method for detecting and repairing anomalies based on gps data

Abstract

The application is applied to abnormal detection and repair of GPS data, and corresponding solving methods are proposed for different types of abnormalities. Different vehicle positioning devices can be regarded as GPS data collection sources. The types of data are various, and the amount of data generated is very large. Due to the influence of geographical environment and bad weather, a large amount of abnormal data exists in the original data. In order to help researchers better utilize the existing data, a framework based on the abnormal detection and repair method of GPS data is designed. In the framework, the maximum disorder sorting algorithm, the near neighbor data filling algorithm, the range constraint and the maximum likelihood estimation of the sliding window are adopted to repair different types of abnormalities, and the path of the trajectory is further improved by analyzing the direction attribute of the data.

Description

I. TECHNICAL FIELD

[0001] The application is applied to the field of GPS anomaly detection and repair, and realizes repair of different types of anomalies by corresponding algorithms in SECONDO through anomaly analysis, thereby improving data quality. II. BACKGROUND

[0002] With the rapid development of science and technology, mobile terminal devices for collecting data are popular, and a large amount of GPS space-time data is generated. These massive data are not all of research value, and there are more or less noises in the data. It is an urgent problem to quickly obtain high-quality data in these data and apply the high-quality data to realize city traffic prediction, road planning, and trajectory analysis.

[0003] The complexity of trajectory data results in many types of anomalies in the data, and it is difficult to comprehensively detect and repair the anomalies by one method, so that it is a difficulty of the algorithm to comprehensively repair the anomalies in the GPS data and improve data quality. High-quality GPS data is not only the basis for the development of traffic management and driverless technology, but also plays a very important role in the development of industry. For example, the collected GPS data are used to automatically calculate the land area of a farmland vehicle, and the quality of the data plays a decisive role in the calculation of the area. Low-quality data will lead to incorrect calculation of the farmland area, causing economic losses of companies and drivers. III. SUMMARY

[0004]

Objectives

[0005] In order to comprehensively detect and repair anomalies in GPS data, improve GPS data quality to provide a good data basis for scientific research, and convert the repaired data into mobile object data to facilitate user analysis of driving trajectories.

[0006]

Technical solutions

[0007] The application realizes a method for comprehensively detecting and repairing anomalies in GPS data, which can repair some abnormal data in the GPS data and thereby improve data quality. Since there are many types of anomalies in the GPS data and different types of abnormal data need to be processed, different anomaly detection and repair algorithms are used in the method. After the high-quality data is repaired, the point data type Point is converted into the mobile object data type MPoint by the extensible mobile object database SECONDO, so as to facilitate analysis of the historical trajectory of the vehicle.

[0008] The main steps are as follows:

[0009] (1) Data preprocessing:

[0010] Firstly, it is judged whether the original GPS data is in order, and the unordered data is sorted according to time sequence. The unordered condition of GPS is caused by signal delay, so the disorder of data is limited, under the constraint condition, in order to improve the running efficiency of the algorithm, the maximum delay algorithm based on heap sorting is designed in the sorting of data, and the repeated data detection and removal work is carried out on the data after sorting.

[0011] (2) Data filling:

[0012] For the data after preprocessing, the missing data and invalid data in the data need to be detected and filled. Invalid data is some data that does not conform to the motion law of trajectory due to device positioning problem, which has no reference value for GPS repair itself, and will affect the analysis of correct data, therefore, we classify the invalid data as missing data for detection and repair, and design a uniform filling method based on the correct data of the near neighbor for filling data, the change of correct data before and after the missing data is used to analyze the state of the data at this time, and the missing data is filled by using the method of uniform acceleration, which is more consistent with the driving law.

[0013] (3) Data repair:

[0014] For the detection and repair of drift type anomaly, the application designs two different methods to detect and repair drift data. For the data deviating from the trajectory, a range constraint method is designed for detection and repair, and for the anomaly within the constraint, a maximum likelihood estimation method based on sliding window is designed to detect and repair small drift anomaly, using the corresponding method to repair the data reduces the modification of the original data and guarantees the integrity of the original data, and in the accurate data repair, the small drift anomaly in the data is repaired more accurately through the analysis of the driving direction.

[0015] (4) Data conversion and visualization:

[0016] After the detection and repair of the anomaly in the GPS data, the clean GPS data obtained is converted into a mobile object data type MPoint through the extensible mobile object database SECONDO, and the history trajectory characteristics of driving can be analyzed through the running trajectory of the mobile point.

[0017]

Advantages

[0018] The GPS anomaly data detection and repair method disclosed by the application comprehensively analyzes the types of anomalies existing in the GPS data, proposes corresponding solving algorithms for different types of anomalies, improves the overall quality of the GPS data, and high-quality GPS data can not only help intelligent traffic prediction, road planning and city management, but also provides a good data basis for the development of big data and unmanned driving technology. IV. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 System overall framework

[0020] Figure 2 Range constraint schematic diagram

[0021] Figure 3 Speed direction analysis intention

[0022] Figure 4 Original data distribution schematic diagram

[0023] Figure 5 Original data collection site data distribution schematic diagram

[0024] Figure 6 Cleaning data distribution diagram V. DETAILED DESCRIPTION

[0025] The abnormality detection and repair method described in the application is realized by multiple algorithms. First, the maximum delay algorithm based on heap sorting is used to sort unordered data and then to deduplicate according to adjacent data. Then, the missing and invalid data are repaired by using the uniform acceleration filling method based on the adjacent data. In the detection and repair of drift data, the range constraint and the maximum likelihood estimation method of the sliding window are combined for detection and repair, and the data are more accurately repaired by analyzing the driving direction. Finally, the data type after cleaning is converted into the mobile object data type by using the extensible mobile object database SECONDO, so as to facilitate the analysis of the driving track. The abnormality detection and repair will be described in detail below in combination with the drawings.

[0026] First, as shown in Figure 1 , the data processing flow of the entire algorithm framework is explained, which is divided into four parts: data preprocessing, data filling, data repair and data conversion.

[0027] (1) Data preprocessing;

[0028] The original GPS data collected by the vehicle positioning system contains many attribute values, such as ID, time, longitude, latitude, speed, driving direction and the like. First, it is necessary to judge whether the data is ordered, i.e. saved in the order of time. Due to the influence of signal delay, the maximum unordered position of the data is limited. For unordered data, a maximum delay method based on heap sorting is designed to sort the data. The number of nodes to be sorted in the heap is mainly set according to the degree of disorder of the data, which improves the sorting efficiency compared with importing the entire data into the heap for sorting. The specific algorithm is as follows:

[0029]

[0030] (2) Data filling;

[0031] In the detection and repair of missing and invalid data, a nearest-neighbor data-based filling method was designed. This method fills in the data based on the data change between the previous and next normal data points through a uniform acceleration process, which better reflects the motion characteristics of vehicle movement. The specific algorithm is shown below:

[0032]

[0033] (3) Data repair;

[0034] The repair of drift data employed a combination of range constraints and sliding window probabilities for detection and repair. For example... Figure 2 The diagram shows three scenarios for detecting and repairing large anomalies in GPS data using range constraints, where p i For outliers that violate range constraints, the first type is the case of consecutive violations, continuing until a normal point with a range intersection with the previous correct point is found, through the intersection of the two nearest boundary points p2 that satisfy the range constraints. r p3 r and the current point p i And the previous point p i-1 The intersection point p1 of the lines connecting them r The constraints are satisfied simultaneously and the distance from p is also satisfied. i The closest point is the most abnormal repair point. The second case indicates that there is one intersection point between the ranges of the two correct data points, and the intersection point is used as the repair point. The third case is when there are two intersection points between the ranges of the correct data points, and the point that satisfies the constraint and is far from p is selected. i The nearest point is used as the repair point. Under the condition of satisfying constraints, the maximum likelihood estimation method based on a sliding window is used to repair the data with abnormally small changes. By statistically analyzing the probability of data changes within the window, the change value with the highest probability is used to repair the data with the smallest change. The directional attribute analysis of the data is as follows: Figure 3 As shown, these correspond to driving in a straight line, curve, right angle, and U-shape, respectively. The position of the normal data is analyzed by using correct data from two points before and after the curve. The distance between the corrected data and the observed data is compared, and a distance threshold of L is set. e If there is a distance error L > L between the repair point and the observation point e The data points are then repaired, and other real-world types can be transformed into the four types mentioned above for processing. For example, an S-shaped road with slow vehicle speeds can be transformed into a curved driving situation. The specific algorithm is shown below:

[0035]

[0036] (4) Data conversion;

[0037] Finally, the repaired clean data is converted from point type Point to mobile object data type MPoint by the SECONDO database. The distribution of the original data is shown in Figure 4 and 5 The distribution of the data after cleaning is shown in Figure 6 Not only can the distribution of the original data and the distribution of the data after repair be compared, but also the running track of the mobile point can be observed to analyze the driving track of the taxi under normal circumstances, which is very helpful for traffic management and prediction.

Claims

1. A method for detecting and repairing GPS anomalies, characterized by detecting and repairing various anomalies in GPS data, providing comprehensive analysis of anomalies in GPS data, and offering corresponding detection and repair algorithms for different types of anomalies. The process consists of the following four parts: (1) Data preprocessing: According to the maximum latency of device storage, the unordered data is sorted using the maximum latency method based on heap sort, and duplicate data is detected and removed from the sorted data; (2) Data filling: Based on the data processed in (1), invalid and missing data in the original GPS data are detected, and the missing data is filled by uniformly accelerating the analysis of the changes between the abnormal data and the normal data around them. (3) Data repair: After filling the data in (2), for drifting abnormal data, the method of combining range constraints and sliding window probability is used to detect and repair. There are three situations for using range constraints to detect and repair large anomalies in GPS data: a. The case of continuous violation of range constraints until a normal point with a range intersection with the previous correct point is found. The point that satisfies the constraints and is closest to the current point is selected from the intersection of the two nearest boundary points that satisfy the range constraints and the intersection of the line connecting the current point and the previous point. b. If there is an intersection point between the ranges of the two correct data points, use the intersection point as the repair point; c. There are two possible intersection points for the correct data range. The point that satisfies both the constraints and is closest to the current point is selected as the repair point. Under the condition of satisfying the constraints, the maximum likelihood estimation method based on a sliding window is used to repair data with small anomalies. By statistically analyzing the probability of data changes within the window, the change value with the highest probability is used to repair the smallest data change. Using these two methods to repair drift data not only reduces the modification to the original data but also ensures the integrity of the original data. Based on this, the GPS data obtained by the velocity method is analyzed by examining the data direction attributes to determine the trajectory path of the GPS acquisition, thereby improving the accuracy of the repair. (4) Data conversion: The high-quality data obtained after repair in (3) is converted to a new data type using the scalable database SECONDO. The data point type Point is converted to the moving object data type MPoint. The movement of the moving point MPoint can intuitively reflect the historical trajectory of the vehicle.

2. The method for detecting and repairing abnormal GPS data according to claim 1, characterized in that: a. The directional attributes of GPS data include straight lines, curves, right angles, and U-shaped travel; b. The position of normal data is analyzed by using correct data from two points before and after the correction, and the distance between the corrected data and the observed data is compared. A corresponding threshold is set, and if the distance error between the corrected point and the observed point is greater than the set threshold, the data point is corrected.

Images