Data processing method and device thereof based on building quick data staging channel

Abstract

The invention provides a data processing method and a device thereof based on building a quick data staging channel. Raw data is identified, classified and filtered by a data filtering device, data with high priority is transferred to the data staging channel, and a scheduling platform utilizes a scheduling strategy to dynamically distribute equipments and call channel data according to the information obtained from the data transmission channel and a data storing device; and the data storing device obtains data source from the scheduling platform and stores the data into a database. The data processing method and the device thereof based on building the quick data staging channel are applicable to various actual services and can build the quick data staging channel, so that the data which needs to be processed prior according to service demands can be separated from massive and miscellaneous data, and meanwhile, data source can be obtained from the scheduling platform actively or passively according to actual situations, and the data is stored in the database in time, so that the flexibility of the system and the validity of data are improved, and service is provided for consumers in a more convenient, quicker and more efficient way.

Description

technical field [0001] The invention belongs to the field of information technology, and relates to a channel-based data processing method and a device thereof, in particular to a highly efficient massive data processing method and a device above 1000 TB (Terabyte) level. Background technique [0002] With the rapid development of network technologies such as the Internet and mobile Internet, the number of Internet users has increased dramatically. In 2008, the digital information created globally reached an astonishing 22-digit number—(39821798684800350000000 bits), which is approximately equal to 40 trillion bits. The total amount of global information will double every two years. In 2011 alone, the total amount of data created and copied in the world reached 1.8ZB. In the face of such a huge amount of data on the Internet, many advanced data processing technologies have been extended. In addition to traditional databases such as oracle, DB2, and Mysql, the development of ...

AI Technical Summary

Problems solved by technology

[0003] In order to meet the needs of business functions and safely and efficiently process massive data, not only can it be solved by using advanced technology, especially when the data scale reaches PB level or above, the data processing capacity will increase with the passage of time and the expansion of storage space. Gradually decline, eventually resulting in a disconnect between business functions and user needs

Images