Data processing method and device

Abstract

The invention provides a data processing method and device. The data processing method comprises the following steps: access processing, synchronously adding an online data record to a corresponding data table under a first catalogue of a distributed file system of a data market periodically according to set access frequency; merge processing which is independent from the access processing, periodically reading the newly added data record in the data table saved under the first catalogue, and merging with the previous data saved in the corresponding data table under a second catalogue of the distributed file system, and recording the current time as the latest update time whenever the data table under the second catalogue is updated; query processing, merging the data table under the second catalogue with the data record newly added after the latest update time when receiving a data query request, and executing the processing of the data query request on the merged data.

Description

Technical field [0001] The present invention relates to the technical field of computers and computer software, in particular to a method and device for data processing. Background technique [0002] Data analysis refers to the process of analyzing a large amount of collected data with appropriate analysis methods, extracting useful information and forming conclusions, and then conducting detailed research and generalization of the data. In practical applications, data analysis can help people make judgments in order to take appropriate actions. [0003] Since the collected data is generally stored in a data warehouse, the data warehouse is the basis for business analysis and decision-making. A data warehouse is a collection of data used by data developers. After the collected data is stored in the data warehouse, data developers then use Hive for data association, analysis and mining of the data stored in the data warehouse. [0004] The data mart is a special data warehouse for ...

AI Technical Summary

Problems solved by technology

Therefore, some requirements with high real-time requirements cannot be met, such as: order risk control requirements (if the behavior of swiping orders is quickly identified, logistics costs can be reduced to a certain extent);

[0009] 2. At present, the delay of the data mart is relatively large. The data obtained by the data analysts of each R&D line is yesterday's data, and there is nowhere to query the data in production;

[0010] 3. Compared with more complex business analysis, the development cost of storm real-time indicator calculation program is relatively large, and it is not easy to maintain

Images