A traffic big data cleaning method based on cloud computing framework

Abstract

The invention discloses a traffic big data cleaning method based on a cloud computing framework. Firstly, the whole data source is scanned, and if missing data exist, filling is performed according to the neighbor quadratic mean value of the dimension where data of the same section are located; then, data with the similar data changing rule are clustered into one class, and the clustering center of the data of the section is obtained; finally, new data are matched with the serial number of the cluster center with the minimum distance, and updating or elimination of abnormal data is performed. Under the cloud computing (Map / Reduce of Hadoop) framework, as for the characteristics of high dimension and mass of traffic data and rapid data updating, the rapid cleaning problem of the massive traffic data is solved by the aid of the parallel computing capability of a clustering system, the similarity characteristics of the traffic data can be rapidly and effectively mined, and the method is used for cleaning abnormal data.

Description

technical field [0001] The invention belongs to the category of intelligent transportation systems, and relates to a method for cleaning traffic big data based on a cloud computing framework. Background technique [0002] Intelligent transportation systems help to improve traffic conditions, and traffic congestion has attracted more and more attention in cities plagued by traffic congestion. Traffic sensors are an important source of data for intelligent transportation systems. However, affected by various factors such as equipment accuracy, equipment failure, and collection environment, abnormal or erroneous data are often collected. This will reduce the accuracy of most applications of intelligent transportation systems (such as traffic state estimation, traffic state prediction). Therefore, it is necessary to clean the traffic data, fill in missing data, eliminate erroneous data, and correct abnormal data. There are many types of traffic sensors currently in use, and t...

AI Technical Summary

Problems solved by technology

[0006] In order to overcome the low processing efficiency of existing traffic data cleaning methods and the inability to apply to big data, the present invention provides a traffic big data cleaning method based on cloud computing framework that improves processing efficiency and is effectively applicable to big data. Under the framework of cloud computing (Hadoop Map / Reduce), aiming at the characteristics of high-dimensional, massive, and fast data update of traffic data, the parallel computing capability of the cluster system is used to solve the problem of rapid cleaning of massive traffic data, which can quickly and effectively Mining traffic data similarity features for cleaning abnormal data

Images