spark-streaming intermediate data partition method, device, computer equipment and storage medium

Abstract

The present application relates to a Spark-Streaming intermediate data partition method, device, computer equipment and storage medium. The method in one embodiment includes: obtaining a plurality of elements in the intermediate data output by the Spark-Streaming map task, and sampling the plurality of elements based on the reservoir sampling algorithm to obtain the element cluster after the sampling process; through time series The prediction method updates the frequency weights corresponding to the elements in the element cluster, and sorts the elements in the updated element cluster according to the preset element order; based on the sorted element cluster, the boundary elements corresponding to the data partition are solved by the dynamic programming method; The elements in the updated element cluster are partitioned according to the boundary elements, so that the sum of the frequency weights corresponding to the elements in the largest partition after partition processing is the smallest.

Description

technical field [0001] The present invention relates to the field of data processing, in particular to a Spark-Streaming intermediate data partition method, device, computer equipment and storage medium. Background technique [0002] With the development of information technology and the rapid growth of network information resources, it is of great significance to process data streams in real time. MapReduce is a standard programming model for processing large-scale data. Apache Spark is an open source implementation of the MapReduce framework. Spark-Streaming is a real-time computing framework built on Spark, which extends Spark's ability to process large-scale streaming data. Spark-Streaming divides the data stream into continuous micro-batch data, and then processes the divided micro-batch data as a series of batch jobs. [0003] Taking a typical Spark batch job processing as an example, the map task reads data, processes the read data according to the user-defined map f...

AI Technical Summary

Problems solved by technology

[0004] When the number of tuples assigned to each partition is different, the amount of data in the partition is also different, which will cause unbalanced reduce task loads for processing these partitions

The completion time of the reduce phase is determined by the slowest task among the parallel reduce tasks. Therefore, when a reduce task is overloaded, it will take a long time to execute, resulting in prolonged job execution time and low job execution efficiency.

That is to say, the traditional intermediate data partition method has the problem of low job execution efficiency

Images