Big data architecture-based data processing method, apparatus and device, and storage medium

Abstract

The invention discloses a big data architecture-based data processing method, apparatus and device, and a storage medium, which are applied to a big data architecture. The architecture is configured with multiple processing functions; and the multiple processing functions have the same core code modules. The method comprises the steps of calling the multiple processing functions to perform presetprocessing on obtained incremental data, thereby obtaining processing results; and storing the processing results in a preset storage layer. According to the method, the apparatus, the device and thestorage medium, the multiple processing functions for processing the incremental data in the architecture have the same core code modules, namely, the multiple processing functions in the architecturehave unified logic layers, so that the multiple processing functions are ensured to be same in logic, the system development and maintenance costs can be reduced, the system maintenance difficulty islowered, and the quality difference of the processing results of processing layers is avoided.

Description

technical field [0001] The present invention relates to the field of big data processing, in particular to a method, device, equipment and storage medium for processing data with a big data architecture. Background technique [0002] In the existing technology, the Lambda architecture is an architecture designed to meet the key characteristics of real-time big data systems, including: high fault tolerance, low latency, and scalability. Moreover, the Lambda architecture integrates offline computing and real-time computing, and integrates a series of architectural principles such as Immunability, read-write separation, and complexity isolation, which can be used to integrate Hadoop, Kafka, Storm, Spark, Hbase and other types of big data components. [0003] The main idea of ​​the Lambda architecture is to divide the big data system architecture into multiple layers, namely the offline processing layer batch layer, the real-time processing layer speed layer, and the service la...

AI Technical Summary

Problems solved by technology

[0004] The main purpose of the present invention is to propose a method, device, device, and storage medium for processing data in a big data architecture, which overcomes the high difficulty, high cost and processing cost of system maintenance caused by the lack of a unified logic layer for processing functions in the big data architecture. There are issues such as quality differences in the results

Images