Intelligent management and control platform operation method based on big data

Abstract

The invention discloses an intelligent management and control platform operation method based on big data, and relates to the technical field of big data. The method comprises a management and controlplatform and the following steps: splitting the script into independent statements by utilizing a script analysis program; obtaining keywords in statements, judging statement types according to the keywords and classifying the statement types into four classes; and respectively creating an object for the script, creating objects for other scripts, a basic object and a static object. According toa script analysis result, two script serial operation mechanisms, namely an intelligent analysis waiting mechanism and a real-time triggering waiting mechanism, are realized, and the intelligent analysis waiting mechanism comprises the following steps: judging the priority of a script object according to the script analysis result. According to the method, two intelligent scheduling mechanisms areused in a mixed mode, the dependency relationship can be achieved at any position of the script, the logic integrity of the script can be guaranteed, the dependency relationship can also be guaranteed, and serial operation of the script is more flexible and efficient.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to an operation method of an intelligent management and control platform based on big data. Background technique [0002] With the rapid development and wide application of big data technology, massive amounts of data are generated every day, and various tasks are carried out for data collection, integration, analysis, mining, etc. The tools, platforms, and operating environments used in these tasks are different. , to ensure the orderly operation of these tasks must have a stable scheduler. [0003] The current market scheduling products have the following defects: 1. At present, the data processing tasks of various businesses are basically processed by the timing scheduling that comes with the system, and the dependencies between tasks are staggered by setting different start times based on approximate time differences. It is easy to cause the previous tasks to be unfinished or ...

AI Technical Summary

Problems solved by technology

[0003] The current market scheduling products have the following defects: 1. At present, the data processing tasks of various businesses are basically processed by the timing scheduling that comes with the system, and the dependencies between tasks are staggered by setting different start times based on approximate time differences. It is easy to cause the previous tasks to be unfinished or fail, and the following tasks to run, and finally run out of wrong analysis results; or the scheduling system used has dependencies, but requires manual configuration, and the configuration is more complicated, which increases the developer's workload. Burden and difficult to maintain; 2. The current scheduling method basically only has feedback information after the task processing is completed, or manually checks the scheduling situation, so that it is impossible to grasp the running status of the task in real time, and once a failure occurs, it is often impossible to deal with it in time ; 3. Due to the large number of big data technologies, the technologies and products used are also different. Tasks such as collection, list, report, and data mining are scattered on different platforms, and task scheduling is also on different platforms. It is easy to manage, and the serial relationship between tasks is not easy to guarantee

Images