Distributed data processing method and system

Abstract

Embodiments of the invention provide a distributed data processing method and system. The method comprises: a shard receives data uploaded by a client for a certain table; the shard stores the data in a storage directory corresponding to the table; when the storage succeeds, the shard sends the data to each of connected stream-mode compute nodes to perform stream-mode computation, so that the data is stored once and then can be simultaneously shared by an offline compute node and a real-time stream-mode compute node without depending on message-oriented middleware. Therefore, the complexity of the system is reduced, a once storage process is reduced compared with a message queue, and storage costs, error probability and processing delay are reduced.

Description

technical field [0001] The present application relates to the technical field of cloud computing, in particular to a distributed data processing method and a distributed data processing system. Background technique [0002] With the rapid development of the Internet and the explosive growth of data volume, cloud computing has been widely used, among which distributed massive data processing is one of the applications of cloud computing. [0003] Distributed massive data processing can be roughly divided into two directions: offline processing and streaming computing. [0004] Offline computing executes query computing on known data sets, such as the offline computing model "MapReduce". [0005] For stream computing, the data is unknown and flows in in real time. When the data flows in, the data is processed according to the defined computing model. [0006] Different computing models determine that offline computing and streaming computing have different requirements for h...

AI Technical Summary

Problems solved by technology

[0013] Although this method provides a unified data landing method for offline computing and streaming computing, the practice of ignoring the differences in computing models also brings some obvious problems

[0014] For offline computing, the data required for computing is often organized in a distributed file system in some form in advance. Therefore, if the message queue is used as the data landing method, the offline computing system also needs an additional data middleware from The data is pulled from the message queue and stored in the distributed file system according to the requirements of offline computing. This not only increases the complexity of the system, but also requires an additional landing process for the data, which increases storage costs and errors. Probability and Latency of Processing

Images