A Spark-Based Anti-skew Data Fragmentation Method

A technology for data sharding and data output, applied in the field of data engines, it can solve the problem of anti-skew mechanism, inability to handle shard skew, etc., and achieve the effect of sharding load

Active Publication Date: 2020-09-01
HUNAN UNIV
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] The purpose of the present invention is to provide a data fragmentation method based on Spark's anti-skew, solve the problem of the

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Spark-Based Anti-skew Data Fragmentation Method
  • A Spark-Based Anti-skew Data Fragmentation Method
  • A Spark-Based Anti-skew Data Fragmentation Method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0098] The present invention will be further described below in conjunction with examples.

[0099] The anti-skew data sharding method based on Spark provided by the present invention can formulate a suitable sharding strategy according to the distribution of intermediate data key clusters (key cluster) and the type of Spark application. Such as figure 1 As shown, the whole process is divided into two parts, the generation of fragmentation strategy and the application of fragmentation strategy. Among them, for different types of Spark applications, different methods are used in policy generation and application. For operations with sorting requirements, use the range sharding algorithm strategy based on key cluster segmentation to generate a weighted boundary array; for other operations, use the hash algorithm strategy based on key cluster redistribution to generate skewed sharding tables and Redistribution policy table. When the sharding strategy is generated, it is only s...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a Spark-based anti-skewing data fragmentation method. The method comprises the steps of A: obtaining key cluster distribution of pre-estimated intermediate data and an application type of Spark, and generating a fragmentation strategy matched with the application type of the Spark based on the obtained key cluster distribution, wherein key clusters are key value pair sets with the same keys, and the fragmentation strategy includes a key cluster reallocation-based hash algorithm strategy and a key cluster segmentation-based range fragmentation algorithm strategy corresponding to application types which do not need to be sorted and needs to be sorted respectively; and B: calculating out a reduce index number of each key value pair in Map output data by utilizing the generated fragmentation strategy, and then sequentially writing the key value pairs into an intermediate data file based on a sequence of the reduce index numbers. Through the method, the problem of ananti-skewing mechanism of the Spark is solved, and the problem that existing Hash method and Range method cannot process fragmentation skewing is solved.

Description

technical field [0001] The invention belongs to the technical field of data engines, and in particular relates to a Spark-based anti-skew data fragmentation method. Background technique [0002] As a fast general engine for big data processing, Spark can divide large-scale data into multiple shards and distribute running tasks among multiple machines in the cluster to process the data. Spark's powerful computing capabilities benefit from its advanced DAG computing engine, which supports asynchronous data flow and memory computing capabilities. [0003] Shuffle is the basis for dividing the stages in the DAG graph, and it is an important and complex process for Spark to process data. It is precisely because the shuffle process well connects the data reading and writing relationship between the front and back stages that multiple machines can process data collaboratively. In this process, the intermediate data output by each Map task will be redistributed to the specified re...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/22
Inventor 唐卓吕葳李肯立李克勤付仲明肖伟
Owner HUNAN UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap