Mass data processing system and method on basis of database

Abstract

The invention provides a mass data processing system and a method on the basis of a database, wherein the system comprises a database, a data collecting device, a data processing device and a data storage device, wherein the database is used for storing original data, the data collecting device is used for extracting data from the database according to set conditions and transmitting the extracted data to the data processing device, the data processing device is used for processing the received data in an odd-even alternation mode according to set dimensionality and transmitting the processed data to the data storage device, and the data storage device is used for storing the received processed data in a classified way. The mass data processing system and the method on the basis of the database disclosed by the invention have the advantages that the processing speed can be accelerated, and the storage space is saved, so the integral performance of the system is improved.

Description

technical field [0001] The present invention relates to a data processing system and method, more specifically, to a database-based massive data processing system and method. Background technique [0002] At present, with the increasing application of databases and the increasing variety of businesses in different fields, it is becoming more and more important to process massive data based on databases (for example, counting the number of active smart cards in a certain time period according to a specific dimension, the amount of data are often at the TB level). [0003] Generally, the basic working principles of existing database-based mass data processing methods are as follows: establish an interface with the database; use middleware for extracting data; create a table on the database side to store certain data extracted from the database Data over a period of time (eg, one day). [0004] However, when the amount of data to be processed is huge (for example, 30 million ...

AI Technical Summary

Problems solved by technology

[0004] However, when the amount of data to be processed is huge (for example, 30 million to 800 million), the above existing technical solutions have the following problems: (1) The data extraction and processing time is too long, so that the system efficiency is reduced and it is difficult to meet the application requirements; (2) The space required for the temporary table of the database is too large, which may cause the system to fail to support or lead to an increase in hardware requirements, and further increase the cost

[0005] In order to solve the aforementioned problems, existing technical solutions usually use some conventional optimization means, such as creating indexes, optimizing database query statements, etc. years) and the number is particularly huge (for example, more than 1 billion), the above-mentioned conventional optimization methods still cannot meet the requirements, that is, the processing time is too long, the space requirement is too large, and the processing cannot even be completed.

Images