Parallel pond sampling dynamic consistency hash partition processing method and system
A partition processing and consistency technology, applied in the direction of resource allocation, multi-programming devices, etc., to improve the overall operating efficiency and the utilization rate of each Reduce node, and solve the load balancing problem
Pending Publication Date: 2022-04-12
CHANGCHUN UNIV OF SCI & TECH
View PDF0 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0006] The purpose of the present invention is to provide a kind of parallel pond sampling dynamic consistent hash partition processing method and system, to solve the load balancing problem of Reduce nodes in a heterogeneous environment, improve the overall operating efficiency of the MapReduce framework in a heterogeneous environment and the utilization of each Reduce node Rate
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0067] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
[0068] The purpose of the present invention is to provide a method and system for processing the dynamic consistent hash partition of parallel pond sampling, to solve the load balancing problem of Reduce nodes in a heterogeneous environment, improve the overall operating efficiency of the MapReduce framework in a heterogeneous environment and the utilization of each Reduce node Rate.
[0069] In order to make the above objects, features and advantages of the p...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a parallel pond sampling dynamic consistency hash partition processing method and system. The method comprises the following steps: carrying out parallel data sampling by adopting a parallel pond sampling algorithm, and calculating the processing speed of each node by utilizing a heartbeat mechanism; and according to the processing speed of each node, performing data distribution by adopting a dynamic consistency hash partitioning strategy, and distributing to-be-processed data to the corresponding Reduce node for data processing. Aiming at the heterogeneity problem in a MapReduce framework, the invention provides a two-stage partitioning strategy, the strategy adopts a parallel pond sampling algorithm to sample data and calculate the processing speed of each node in the first stage, adopts a dynamic consistency hash partitioning strategy to carry out data distribution in the second stage, and sets virtual nodes according to the node processing speed, so that the data distribution efficiency is improved. And the nodes with higher speed can process more data, so that the overall operation efficiency of the MapReduce framework in the heterogeneous environment and the utilization rate of each Reduce node are improved, and the load balancing problem of the Reduce nodes in the heterogeneous environment is solved.
Description
technical field [0001] The invention relates to the technical field of distributed parallel computing, in particular to a method and system for processing dynamic consistent hash partitions of parallel pond sampling. Background technique [0002] The heterogeneity of the MapReduce framework is an important issue that affects the performance of the framework. In a heterogeneous environment, due to the computing power of the Reduce node and the network delay, the idea of balanced allocation is not applicable to the heterogeneous environment. In response to this kind of problem, many scholars at home and abroad have proposed a variety of strategies and methods, which can be summarized as using partition strategy, task scheduling algorithm and other aspects to deal with it. [0003] The performance of each MapReduce node in a heterogeneous environment reduces the efficiency of the framework. The most direct factor is that the heterogeneity of the nodes is not considered when d...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F9/50
Inventor 杨迪赵家伟王鹏李松江任志鹏董明
Owner CHANGCHUN UNIV OF SCI & TECH