Method, device and equipment for processing data skew and storage medium

Abstract

The invention relates to the field of big data, and provides a method, device and equipment for processing data skew and a storage medium. The method comprises the steps: presetting the capacity of ato-be-stored space, and setting the data type and target number in the to-be-stored space; partitioning the to-be-stored space according to a preset rule based on the capacity and the data types to obtain the size and the number of sub-storage spaces corresponding to each data type; according to the size of each sub-storage space and the target number corresponding to the data type, determining the size and the number of target storage spaces in each partition through a partition rule; setting a random number of each partition according to the number of the target storage spaces in each partition; marking the random number and a preset judgment condition on each part of data in the to-be-stored space; and analyzing the content marked on each part of data in the to-be-stored space through the random grouping function so as to store the data corresponding to each data type into the target to-be-stored space. By adopting the scheme, the problem of data skew processing can be effectively solved.

Description

technical field [0001] The present application relates to the field of data processing, and in particular to a method, device, device and storage medium for processing data skew. Background technique [0002] With the rapid development of technologies such as the Internet of Things, cloud computing, and network bandwidth, big data computing is widely used. In big data computing, the transmission or storage or processing of massive data information often leads to data skew. In the existing data warehouse tool Hive, there are no adjustable parameters or callable functions to directly solve the problem of data skew when reading data, so that Hive and other big data computing engines are reading When the Hive table data is skewed, it cannot be processed in a timely and effective manner, resulting in the failure of the entire task to be completed within the stipulated time limit, thus failing to meet business requirements. [0003] In the current data skew processing, by detect...

AI Technical Summary

Problems solved by technology

[0004] Since the merged file needs to be split when obtaining a single original input file, the increase in data processing resources will affect the running time, and the increase in Map data will lead to an increase in data processing time, which will affect the operating efficiency. Therefore, it cannot be effectively solved. Data Skew Processing Problem

Images