Distributed computing system and method

Abstract

The invention discloses a distributed computing system and method. The system comprises clusters, a global meta data server, a trans-cluster data copy server and a data access distribution server; the clusters carry out distributed computing; the global meta data server stores and manages a multi-cluster data duplicate distribution view as well as a multi-cluster topology structure view; according to a copying instruction, the trans-cluster data copy server copies specific data into a specific cluster corresponding to the copying instruction; and according to a data using request of the cluster, the multi-cluster data duplicate distribution view and the multi-cluster topology structure view, the data access distribution server distributes position information for target data, and generates the copying instruction according to the position information of the target data, so that the trans-cluster data copy server copies a target data duplicate of the target data into the cluster corresponding to the data using request. The system ensures that service data can be obtained within a limited waiting time range in a computing task when service data duplicates are distributed dynamically in multiple positions in a multi-cluster environment limited by network conditions.

Description

technical field [0001] The present application relates to the technical field of the Internet, and in particular to a distributed computing system and method. Background technique [0002] At present, the offline data processing methods of mainstream data business companies are mostly distributed data computing tasks running on clusters. With the continuous expansion of data scale, multi-cluster collaborative work has become the mainstream way to process large-scale offline data. In the scenario of cross-regional multi-cluster collaboration with limited network bandwidth, how to efficiently access data between clusters has become an important issue. [0003] In related technologies, the multi-cluster data access method usually adopts the following method when computing tasks need to access data across clusters: directly read data across clusters through a network directly connected between clusters. [0004] However, the existing method of directly accessing the data locat...

AI Technical Summary

Problems solved by technology

[0004] However, the existing method of directly accessing the data location through metadata information has problems in the following three aspects: (1) When a certain piece of hot data is accessed by a large number of computing tasks across clusters and regions at the same time, the network bandwidth will be greatly reduced. become a bottleneck, leading to problems such as access delays and communication quality degradation. Difficult to achieve; (3) Higher requirements for network stability, such as the network must be available at any time, and long-distance links are difficult to achieve

[0005] Therefore, the existing method of directly accessing the location of data through metadata information cannot be applied to complex network environment scenarios with multiple clusters across regions

Images