A method and device for data processing

Abstract

The invention provides a data processing method and device. The data processing method comprises the following steps: access processing, synchronously adding an online data record to a corresponding data table under a first catalogue of a distributed file system of a data market periodically according to set access frequency; merge processing which is independent from the access processing, periodically reading the newly added data record in the data table saved under the first catalogue, and merging with the previous data saved in the corresponding data table under a second catalogue of the distributed file system, and recording the current time as the latest update time whenever the data table under the second catalogue is updated; query processing, merging the data table under the second catalogue with the data record newly added after the latest update time when receiving a data query request, and executing the processing of the data query request on the merged data.

Description

technical field [0001] The present invention relates to the technical field of computers and computer software, in particular to a data processing method and device. Background technique [0002] Data analysis refers to the process of analyzing a large amount of collected data with appropriate analytical methods, extracting useful information and forming conclusions to study and summarize the data in detail. In practical terms, data analysis can help people make judgments so that appropriate actions can be taken. [0003] Since the collected data is generally stored in the data warehouse, the data warehouse is the basis for business analysis and decision-making. A data warehouse is a collection of data used by data developers. After the collected data is stored in the data warehouse, the data R&D personnel use Hive to perform data association, analysis and mining on the data stored in the data warehouse. [0004] Data mart is a special data warehouse for query and analysi...

AI Technical Summary

Problems solved by technology

Therefore, some requirements with high real-time requirements cannot be met, such as: order risk control requirements (if the behavior of swiping orders is quickly identified, logistics costs can be reduced to a certain extent);

[0009] 2. At present, the delay of the data mart is relatively large. The data obtained by the data analysts of each R&D line is yesterday's data, and there is nowhere to query the data in production;

[0010] 3. Compared with more complex business analysis, the development cost of storm real-time indicator calculation program is relatively large, and it is not easy to maintain

Images