A Spark Procrastination Task Diagnosis Method Based on Statistical Analysis

Abstract

The invention relates to a Spark procrastination task diagnosis method based on statistical analysis, which monitors data centers to execute tasks in parallel, and deduces the reason of task executiondelay from the monitoring data. In the same phase, the same task in different nodes is monitored, the monitoring data of the physical server is collected, the Spark log file is analyzed to obtain the monitoring data of task execution. Task execution time is compared to detect delayed tasks, and the deviation of delayed tasks from the eigenvalues of normal tasks is analyzed to locate abnormal features, so as to diagnose the causes of task delays.

Description

Technical field [0001] The invention relates to a Spark delay task diagnosis method based on statistical analysis, and belongs to the technical field of software. Background technique [0002] Spark is a parallel data processing model, and Stragglers are extremely slow tasks in Spark's data processing jobs, which delay the completion of the entire job, and are common in data centers with multiple jobs. The data center decomposes data processing computing tasks into many tasks, which are executed in parallel on multiple machines, and the results are aggregated when the last task is completed. Stragglers threatens the parallel computing performance of the data center, and its impact will vary with the number of tasks and system performance. Increase in scale. Studies have shown that in the Google data center, delayed tasks cause 20% of the work to complete more than 1.5 times. In Facebook and Microsoft's data centers, Straggler increased the average time to complete a job by 47% ...

AI Technical Summary

Problems solved by technology

At the same time, the existing Straggler migration strategy only deals with the symptoms that occur without diagnosing the cause of the problem

Although generating a large number of task execution traces in the data center can help diagnose Stragglers, it is very difficult to obtain and explain the cause of the problem from a complex data set

Existing diagnostics rely on systematic domain knowledge and applied best practices and are difficult to scale (Ganesh Ananthanarayanan, Ali Ghodsi, Scott Shenker, and Ion Stoica. 2013. Effective Straggler Mitigation: Attack of the Clones. In Proceedings of the 10th USENIXConference on Networked Systems Design and Implementation.185–198.) ( Jeffrey Dean and Luiz André Barroso. 2013. The tail at scale. ACM Communication. 56(2): 74–80, 2013.)

Images