Cross-multi-data-center data distributed processing acceleration method and system

A distributed processing and multi-data technology, applied in the field of data analysis, can solve problems such as insufficient consideration of site heterogeneity
CN112532464AActive Publication Date: 2021-03-19NAT UNIV OF DEFENSE TECH
6 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN Β· China
Current Assignee / Owner
NAT UNIV OF DEFENSE TECH
Publication Date
2021-03-19

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention provides a cross-multi-data-center data distributed processing acceleration method. According to the method, each station can execute the corresponding calculation task as long as obtaining the required input data. And the processes of input data loading, map calculation, buffer transmission and reduce calculation of each site do not need to wait for the previous process of other sites to complete the corresponding operation. Meanwhile, accurate calculation time estimation is provided, the method adapts to the dynamic wide area network bandwidth to improve the practicability of the SDTP, and the response time of operation can be greatly shortened. The invention further provides a cross-multi-data-center data distributed processing acceleration system, corresponding to the method, the network and computing resources of the cross-regional distribution sites can be fully used, and therefore the cross-regional distribution data can be effectively analyzed without waiting forthe bottleneck site of the previous stage to complete the corresponding data transmission or computing task.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the field of data analysis, and specifically discloses a data distributed processing acceleration method across multiple data centers and a system thereof. Background technique

[0002] Cloud providers such as Google, Amazon, and Alibaba have deployed data centers around the world to provide instant services. These services generate large amounts of data globally, including transaction data, user logs, and performance logs, among others. Mining these geographically distributed data (also known as wide-area analytics) is critical for business recommendations, anonymous detection, performance upgrades, and system maintenance, among others. A distributed computing framework such as Map-Reduce is usually implemented to mine such massive datasets. The main challenge of this computing method is the heterogeneity of hardware resources among geographically distributed sites, mainly including computing, uplink bandwidth and downlink b...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More