Cluster computing resource scheduling method, device and equipment and medium

Abstract

The embodiment of the invention discloses a cluster computing resource scheduling method and device based on a neural network algorithm, equipment and a medium. The method comprises the following steps: submitting a task; preprocessing the task, and obtaining preprocessing information; extracting data features, task features and cluster computing resource features; calculating pre-processing information, data characteristics, task characteristics and cluster computing resource characteristics by using a neural network algorithm, and generating resource allocation information; running a task according to the resource distribution information, and obtaining task resource monitoring, log feature extraction and accuracy verification; according to task resource monitoring, log feature extraction and accuracy verification, a neural network model is trained; and submitting the task again, repeating the above operations for many times, and outputting a final cluster computing resource scheduling scheme after the trained neural network model performs calculation. According to the embodiment of the invention, an intelligent and automatic method is realized, the equipment utilization rate andthe data analysis efficiency are improved, and the energy consumption of the data center is reduced.

Description

technical field [0001] The present invention relates to the fields of software, communication, Internet, and big data industries, specifically to the field of comprehensive application technology of artificial intelligence technology and big data technology, and in particular to a cluster computing resource scheduling method, device, equipment, and medium based on a neural network algorithm . Background technique [0002] In a production cluster based on Hadoop technology, theoretically, the task’s request for computing resources should be satisfied in a timely manner, but in reality, the cluster’s computing resources are often limited, and when tasks need computing resources, they often need to wait in line. Existing It is difficult for technology to find a strategy that can satisfy all application scenarios. [0003] Existing technical solutions generally use the scheduler and policies that come with Hadoop technology, and system operation and maintenance managers manuall...

AI Technical Summary

Problems solved by technology

[0005] Embodiments of the present invention provide a cluster computing resource scheduling method, device, device, and medium based on a neural network algorithm, which are used to solve the problem of inflexible task resource allocation, low task operation efficiency, and difficult data center operation and maintenance in the prior art. As well as the problems of manual operation, in order to achieve intelligent, dynamic, automatic, real-time resource management and scheduling

Images