Unlock instant, AI-driven research and patent intelligence for your innovation.

A Fast Processing Method of Massive Traffic Data Based on Distributed System

A distributed system and traffic data technology, applied in the field of rapid processing of massive traffic data, can solve the problems of large traffic data volume, fast query speed and high efficiency, and achieve the effect of efficient query and improved data processing efficiency

Active Publication Date: 2019-05-14
ENC DATA SERVICE CO LTD
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] Aiming at the above-mentioned technical problems of large amount of traffic data and time-consuming processing, the present invention proposes a method for fast processing of massive traffic data based on a distributed system, which has fast query speed and high efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Fast Processing Method of Massive Traffic Data Based on Distributed System
  • A Fast Processing Method of Massive Traffic Data Based on Distributed System
  • A Fast Processing Method of Massive Traffic Data Based on Distributed System

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0033] The present invention will be described in detail below in conjunction with the accompanying drawings. The following examples do not limit the invention. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be imagined by those skilled in the art are all included in the present invention.

[0034] figure 1 It is a flow chart of data processing according to the method of the present invention, including a data preprocessing process (steps S1-S2) and a data request processing process (steps S3-S6).

[0035] The data preprocessing process will be described in detail below. Preprocessing includes dividing the original data according to time and space, adding time and space identification fields, and storing them on the file system according to rules.

[0036] Firstly, the mass traffic data is divided into blocks in terms of time and space, and the divided traffic data has a time attribute identifier and a space attribute i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a massive traffic data rapid processing method based on a distributed system. The massive traffic data rapid processing system comprises the following steps: a, carrying out partitioning on massive traffic data in time and space, wherein the partitioned traffic data has a time attribute identifier and a space attribute identifier; b, storing a certain quantity of the commonly-used traffic data in a memory by fine granularity, and storing traffic data, except the traffic data stored in the memory, in a file system by coarse granularity; c, judging whether the requested traffic data is in the memory when a processing program requests to process the data; d, if the requested traffic data is stored in the memory, directly processing corresponding data; and e, if the requested traffic data is stored in the file system, reading the traffic data into the memory from the file system according to file information, and then processing the traffic data. The method provided by the invention is high in inquiry speed and high in efficiency.

Description

technical field [0001] The invention relates to the fields of a distributed memory computing platform Spark and a distributed file system HDFS, in particular to a method for quickly processing massive traffic data based on a distributed system. Background technique [0002] Spark is an in-memory computing framework that provides MapReduce operations similar to Hadoop (distributed system basic framework), but the intermediate results are not stored in the file system, but in memory. Compared with Hadoop, the calculation speed is greatly improved. . [0003] The full name of HDFS is Hadoop Distributed Filesystem, which is the flagship file system of Hadoop. Its idea comes from the Google File System (Google File System, GFS), and is suitable for the access mode of writing once and reading many times, satisfying the application scenarios of urban multi-source data. It is a distributed file system suitable for storing large files and can be used as a data source for Hadoop and...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/182G06F16/172
CPCG06F16/172G06F16/182
Inventor 张善海熊贵喜蔡朝辉杜博文凌萍谢志普
Owner ENC DATA SERVICE CO LTD