Big data platform resource preemption test method and system, terminal and storage medium

Abstract

The invention provides a big data platform resource preemption test method and system, a terminal and a storage medium. The method comprises the following steps: respectively setting a first task queue resource proportion and a second task queue resource proportion; setting task data and constructing a query task for the task data; under the cluster no-load condition, submitting the query task toa first task queue and a second task queue to obtain first processing time and second processing time required for task completion, and ensuring that the first processing time and the second processing time are equal; submitting a query task to the first task queue, submitting the query task to the second task queue when the query task of the first task queue is executed to a preset progress, andobtaining third time when the first task queue completes the task and fourth time when the second task queue completes the task; and monitoring a resource preemption event according to a comparison relationship among the first time, the second time, the third time, the fourth time, the first task queue resource proportion and the second task queue resource proportion.

Description

technical field [0001] The present invention relates to the technical field of big data service platforms, in particular to a method, system, terminal and storage medium for testing resource preemption of big data platforms. Background technique [0002] The big data component YARN has three scheduling modes. Fair Scheduler is one of the widely applicable scheduling modes, and this mode supports resource preemption, which not only ensures that fewer tasks are processed quickly, but also ensures fair scheduling when tasks are concurrent. , so the resource preemption function is an important function of this component. In the YARN component test of the big data platform, resource scheduling is implemented internally, and the occurrence of resource preemption through code analysis is very complicated. Therefore, there is a lack of testing methods for resource preemption, and there are few test solutions that can be used for reference. Contents of the invention [0003] In vi...

AI Technical Summary

Problems solved by technology

In the YARN component test of the big data platform, resource scheduling is implemented internally, and the occurrence of resource preemption through code analysis is very complicated, so there is a lack of test methods for resource preemption, and there are few test plans that can be used for reference.

Images