Data inclination prediction method and device

Abstract

The invention provides a data skew prediction method and device, and the method comprises the steps: responding to a data skew prediction instruction, and determining a to-be-executed task corresponding to the data skew prediction instruction; acquiring current running state information of the node cluster and data volume information of a source system, wherein the source system is to be used forproviding task data corresponding to the to-be-executed task, and the node cluster is to be used for processing the task data corresponding to the to-be-executed task; generating a prediction parameter corresponding to the to-be-executed task based on the task operator, the running state information and the data volume information of the to-be-executed task; and inputting the prediction parametersinto a preset data inclination prediction model to obtain a data inclination prediction result corresponding to the to-be-executed task. By applying the method provided by the invention, the data inclination prediction result corresponding to the to-be-executed task can be obtained before the to-be-executed task is executed, and data inclination in the task execution process can be avoided, so that other services using the cluster are prevented from being influenced.

Description

technical field [0001] The invention relates to the technical field of computer applications, in particular to a method and device for predicting data skew. Background technique [0002] With the development of computer technology, big data processing technology has also been popularized in the face of ever-increasing massive data. However, many problems have also appeared in the process of processing large amounts of data. The most common problems in the process. [0003] Data skew refers to the fact that when the cluster is executing large data tasks, due to insufficient dispersion of cached data, a large amount of data is concentrated on one or several data nodes in the cluster; this will make the processing speed of these data nodes slower than Average processing speed, slowing down the entire task execution process. If the skewed data exceeds the memory limit set by the data node itself, the data node will crash. [0004] In the prior art, it is usually found that th...

AI Technical Summary

Problems solved by technology

[0004] In the prior art, it is usually found that there is data skew in the cluster when it is detected that the executed task takes too long to run or an out of memory (Out Of Memory, OOM) exception occurs during the execution of a big data task. After the skew occurs, it takes a lot of time to solve the data skew problem, which affects other businesses currently using the cluster

Images