A visual scheduling method for computing tasks based on artificial intelligence

Abstract

The present invention provides a method for visual scheduling of computing tasks based on artificial intelligence, including: when a computing task is created, the system automatically recommends the upstream tasks and executable time on which the task depends; when scheduling computing tasks, the scheduling priority is determined according to the in-degree and out-degree of the task, and the computing tasks to be scheduled are sorted and scheduled in parallel using the breadth-first parallel scheduling algorithm; the computing task visualization uses the critical path algorithm to display the critical path of the current computing task node to avoid visual interference by computing task nodes on other paths; the computing task running log analysis performs corpus segmentation on the error log, and uses the TF-IDF algorithm to perform word frequency analysis, and archives and classifies the error types for unified processing. The present invention introduces an artificial intelligence algorithm, proposes reasonable prompts for the arrangement of computing tasks according to the correlation analysis algorithm, and uses the TF-IDF algorithm to analyze the running log of the computing task, thereby improving the efficiency of computing task scheduling.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a visualization scheduling method for computing tasks based on artificial intelligence. Background Art

[0002] With the advent of the big data era, the demand for data in all walks of life is becoming more and more vigorous, and these data often need to be processed before they can be fully utilized. Data processing is the operation process of multiple computing tasks. These computing tasks are often triggered on time by the scheduling device in the data center. However, when there are many computing tasks, this triggering method is prone to form peaks and troughs, resulting in insufficient resource utilization. Therefore, the intelligent scheduling and visual management of computing tasks have become extremely important. It is urgent to propose a visual scheduling method based on artificial intelligence to solve these problems. Summary of the invention

[0003] The purpose of the present invention is to provide a method for visual scheduling of computing tasks based on artificial intelligence, introduce artificial intelligence algorithms, make reasonable suggestions for the scheduling of computing tasks according to the correlation analysis algorithm, and use the TF-IDF algorithm to analyze the operation logs of computing tasks, organize error information, improve the efficiency of fault handling, and optimize the existing computing task scheduling strategies and management methods.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A method for visual scheduling of computing tasks based on artificial intelligence includes the following steps:

[0006] S1, computing task creation: When a computing task is created, the data middle-end system recommends the upstream tasks and executable time that the task depends on through a correlation algorithm;

[0007] S2, computing task scheduling: When to schedule a computing task requires comprehensive consideration of multiple factors, including whether the upstream task is successfully executed and whether it is the specified execution time. After it is schedulable, there will be multiple tasks that need to be scheduled at the same time. At this time, the scheduling priority is determined according to the in-degree and out-degree of the task, and the breadth-first parallel scheduling algorithm is used to sort and schedule the computing tasks to be scheduled in parallel.

[0008] S3, computing task visualization: Use the critical path algorithm to display the critical path of the current computing task node, avoid visual interference from computing task nodes on other paths, select the computing task node, and display the timeline of the computing task node, including the trigger time, running start time and running end time;

[0009] S4, computing task operation log analysis: perform corpus segmentation on the error logs in the computing tasks, use the TF-IDF algorithm to perform word frequency analysis, archive and classify the error types, and process them uniformly.

[0010] Furthermore, in step S1, a computing task is created, and the data middle platform system automatically recommends the upstream tasks and executable time that the task depends on, including:

[0011] S101, the user fills in the relevant information of the computing task, including description and task tags, through the task creation guide page;

[0012] S102, performing natural language processing on the task description information, and performing correlation analysis in combination with the task tags, and finally outputting a list of recommended dependent tasks for the computing task;

[0013] S103, obtaining the scheduling trigger time and running time of the dependent tasks selected by the computing task according to the dependent task recommendation list, and determining the time period for triggering the running of the task;

[0014] S104, query the task list of the time period, and the system recommends the trigger time point of the task, specifically including: when the system recommends, first determine the time period of the task based on the time points of successful operation of all upstream tasks of the task in the past 7 days, then count the task distribution within the time period, and recommend the time point with the least tasks as the trigger time point of the task.

[0015] Furthermore, in step S2, the scheduling priority is determined according to the in-degree and out-degree of the task, and the computing tasks to be scheduled are sorted and scheduled in parallel using the breadth-first parallel scheduling algorithm, which specifically includes:

[0016] S201, forming a DAG directed acyclic graph from related computing tasks, with each computing task being a node in the DAG directed acyclic graph;

[0017] S202, based on the DAG directed acyclic graph, between nodes of the same level, weights are determined according to the in-degree and out-degree of the nodes, and scheduling priorities are determined according to the weights;

[0018] S203, improving the breadth-first traversal algorithm so that it can be executed in parallel, obtaining a breadth-first parallel scheduling algorithm, using the breadth-first parallel scheduling algorithm to traverse the computing tasks to be scheduled, and performing parallel scheduling according to the scheduling priority.

[0019] Furthermore, the breadth-first parallel scheduling algorithm is:

[0020] When traversing the DAG in breadth, the computing task nodes in each layer are placed in array A, and the computing tasks are triggered asynchronously by traversing array A so that the computing tasks can run in parallel. After a computing task is successfully executed, its subtask nodes are placed in array x, and then the array is traversed and computing tasks are triggered asynchronously. The above process is repeated until all computing task nodes in the DAG graph are traversed.

[0021] Furthermore, in step S203, the weight is determined according to the in-degree and out-degree of the computing task node, and the scheduling priority is determined according to the weight, specifically:

[0022] The computing task nodes with fewer in-degrees and more out-degrees have greater weights and higher scheduling priorities.

[0023] The calculation formula is W i =in i *0.2+out i *0.8, where W i is the weight of computing task node i, in i is the in-degree of the computation task node i, out i is the out-degree of the computation task node i.

[0024] Furthermore, in step S4, the computing task operation log is analyzed: the error log in the computing task is segmented, and the word frequency analysis is performed using the TF-IDF algorithm, and the error types are archived and classified for unified processing, specifically including:

[0025] S401, performing log level filtering on the task running log;

[0026] S402, performing corpus segmentation on the logs at the warning level and the error level;

[0027] S403, using the TF-IDF algorithm to perform word frequency analysis on the warning level logs and the error level logs, and classify the error types.

[0028] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the artificial intelligence-based computing task visualization scheduling method provided by the present invention creates a computing task life cycle in the data center, which includes computing task creation, computing task scheduling, computing task visualization and computing task operation log analysis; introduces an artificial intelligence algorithm, performs natural language processing on the task description information according to the correlation analysis algorithm, and performs correlation analysis in combination with the task tags, so as to provide reasonable suggestions for the scheduling of computing tasks, and uses the TF-IDF algorithm to analyze the operation log of the computing task, organize the error information, and improve the efficiency of fault handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 The figure is a flow chart of the method for visual scheduling of computing tasks based on artificial intelligence of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The purpose of the present invention is to provide a method for visual scheduling of computing tasks based on artificial intelligence, introduce artificial intelligence algorithms, make reasonable suggestions for the scheduling of computing tasks according to the correlation analysis algorithm, and use the TF-IDF algorithm to analyze the operation logs of computing tasks, organize error information, improve the efficiency of fault handling, and optimize the existing computing task scheduling strategies and management methods.

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] like Figure 1 As shown, the computing task visualization scheduling method based on artificial intelligence provided by the present invention includes the following steps:

[0035] S1, computing task creation: Create a computing task and automatically recommend the upstream tasks and executable time that the task depends on through the data middle platform system;

[0036] S2, computing task scheduling: When to schedule a computing task requires comprehensive consideration of multiple factors, including whether the upstream task is successfully executed and whether it is the specified execution time. After it is schedulable, there will be multiple tasks that need to be scheduled at the same time. At this time, the scheduling priority is determined according to the in-degree and out-degree of the task, and the breadth-first parallel scheduling algorithm is used to sort and schedule the computing tasks to be scheduled in parallel.

[0037] S3, computing task visualization: Use the critical path algorithm to display the critical path of the current computing task node, avoid displaying too many computing task nodes on other paths to cause visual interference, and easily find the operation bottleneck, optimize the task flow, select the task computing task node, and display the timeline of the computing task node, including the trigger time, operation start time and operation end time;

[0038] S4, computing task operation log analysis: perform corpus segmentation on the error logs in the computing tasks, use the TF-IDF algorithm to perform word frequency analysis, archive and classify the error types, and process them uniformly.

[0039] In step S1, a computing task is created, and the data middle platform system automatically recommends the upstream tasks and executable time that the task depends on, including:

[0040] S101, the user fills in the relevant information of the computing task, including description and task tags, through the task creation guide page;

[0041] S102, performing natural language processing on the task description information, and performing correlation analysis in combination with the task tags, and finally outputting a list of recommended dependent tasks for the computing task;

[0042] S103, obtaining the scheduling trigger time and running time of the dependent tasks selected by the computing task according to the dependent task recommendation list, and determining the time period for triggering the running of the task;

[0043] S104, query the task list of the time period, and the system recommends the trigger time point of the task. When the system recommends, it first determines the time period of the task based on the successful running time points of all upstream tasks of the task in the past 7 days, and then counts the distribution of tasks in the time period, and recommends the time point with the least tasks as the trigger time point of the task.

[0044] In step S2, computing task scheduling: determining the scheduling priority according to the in-degree and out-degree of the task, and using the breadth-first parallel scheduling algorithm to sort and parallel schedule the computing tasks to be scheduled, specifically including:

[0045] S201, forming a DAG directed acyclic graph from related computing tasks, with each computing task being a node in the DAG directed acyclic graph;

[0046] S202, based on the DAG directed acyclic graph, among the computing task nodes of the same level, the weight is determined according to the in-degree and out-degree of the computing task nodes, and the scheduling priority is determined according to the weight; the computing task node with fewer in-degrees and more out-degrees has a greater weight and a higher scheduling priority; the calculation formula is W i =in i *0.2+out i*0.8, where W i is the weight of computing task node i, in i is the in-degree of the computation task node i, out i is the out-degree of the computation task node i;

[0047] S203, improving the breadth-first traversal algorithm so that it can be traversed in parallel, obtaining a breadth-first parallel scheduling algorithm, using the breadth-first parallel scheduling algorithm to traverse the computing tasks to be scheduled, and performing parallel scheduling according to the scheduling priority.

[0048] The breadth-first parallel scheduling algorithm is:

[0049] When traversing the DAG in breadth, the computing task nodes in each layer are placed in array A, and the computing tasks are triggered asynchronously by traversing array A so that the computing tasks can run in parallel. After a computing task is successfully executed, its subtask nodes are placed in array x, and then the array is traversed and computing tasks are triggered asynchronously. The above process is repeated until all computing task nodes in the DAG graph are traversed.

[0050] Among them, in the step S4, the computing task running log analysis: the error log in the computing task is segmented, and the word frequency analysis is performed using the TF-IDF algorithm, and the error types are archived and classified for unified processing, which specifically includes:

[0051] S401, performing log level filtering on the task running log;

[0052] S402, performing corpus segmentation on the logs at the warning level and the error level;

[0053] S403, using the TF-IDF algorithm to perform word frequency analysis on the warning level logs and the error level logs, and classify the error types.

[0054] In summary, the computing task visualization scheduling method based on artificial intelligence provided by the present invention creates a computing task life cycle in the data center, which is computing task creation, computing task scheduling, computing task visualization and computing task running log analysis; when a computing task is created, the system automatically recommends the upstream tasks and executable time on which the task depends; when scheduling computing tasks, the scheduling priority is determined according to the in-degree and out-degree of the task, and the computing tasks to be scheduled are sorted and scheduled in parallel using the breadth parallel priority scheduling algorithm; computing task visualization uses the critical path algorithm to display the critical path of the current computing task node to avoid visual interference from other computing task nodes; computing task running log analysis performs corpus segmentation on the error log, and uses the TF-IDF algorithm for word frequency analysis, archives and classifies the error types, and processes them uniformly. The present invention introduces an artificial intelligence algorithm, proposes reasonable prompts for the arrangement of computing tasks according to the correlation analysis algorithm, and uses the TF-IDF algorithm to analyze the running log of the computing task, organizes the error information, and improves the efficiency of fault handling.

[0055] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for visual scheduling of computing tasks based on artificial intelligence, applied to a data middle-end system, characterized in that: The following steps are involved: S1, computing task creation: When a computing task is created, the data middle-end system recommends the upstream tasks and executable time that the task depends on through a correlation algorithm; S2, computing task scheduling: When to schedule a computing task requires comprehensive consideration of multiple factors, including whether the upstream task is successfully executed and whether it is the specified execution time; After being schedulable, there will be multiple tasks that need to be scheduled at the same time. At this time, the scheduling priority is determined according to the in-degree and out-degree of the task, and the breadth-first parallel scheduling algorithm is used to sort and schedule the computing tasks to be scheduled in parallel; S3, computing task visualization: Use the critical path algorithm to display the critical path where the current computing task is located, avoid visual interference by displaying computing tasks on other paths, select a computing task, and display the timeline of the computing task, including the trigger time, running start time, and running end time; S4, computing task operation log analysis: perform corpus segmentation on the error logs in the computing tasks, use the TF-IDF algorithm to perform word frequency analysis, archive and classify the error types, and process them uniformly; In step S1, a computing task is created, and the data middle platform system automatically recommends the upstream tasks and executable time that the task depends on, including: S101, the user fills in the relevant information of the computing task, including description and task tags, through the task creation guide page; S102, performing natural language processing on the task description information, and performing correlation analysis in combination with the task tags, and finally outputting a list of recommended dependent tasks for the computing task; S103, obtaining the scheduling trigger time and running time of the dependent tasks selected by the computing task according to the dependent task recommendation list, and determining the time period for triggering the running of the task; S104, query the task list of the time period, and the system recommends the trigger time point of the task, specifically including: when the system recommends, first determine the time period of the task based on the time points of successful operation of all upstream tasks of the task in the past 7 days, then count the task distribution within the time period, and recommend the time point with the least tasks as the trigger time point of the task.

2. The method for visual scheduling of computing tasks based on artificial intelligence according to claim 1, characterized in that: In step S2, the scheduling priority is determined according to the in-degree and out-degree of the task, and the computing tasks to be scheduled are sorted and scheduled in parallel using the breadth-first parallel scheduling algorithm, which specifically includes: S201, forming a DAG directed acyclic graph from related computing tasks, with each computing task being a node in the DAG directed acyclic graph; S202, based on the DAG directed acyclic graph, between nodes of the same level, weights are determined according to the in-degree and out-degree of the nodes, and scheduling priorities are determined according to the weights; S203, improving the breadth-first traversal algorithm so that it can be executed in parallel, obtaining a breadth-first parallel scheduling algorithm, using the breadth-first parallel scheduling algorithm to traverse the computing tasks to be scheduled, and performing parallel scheduling according to the scheduling priority.

3. The method for visual scheduling of computing tasks based on artificial intelligence according to claim 2, characterized in that: The breadth-first parallel scheduling algorithm is: When traversing the DAG in breadth, the computing task nodes in each layer are placed in array A, and the computing tasks are triggered asynchronously by traversing array A so that the computing tasks can run in parallel. After a computing task is successfully executed, its subtask nodes are placed in array x, and then the array is traversed and computing tasks are triggered asynchronously. The above process is repeated until all computing task nodes in the DAG graph are traversed.

4. The method for visual scheduling of computing tasks based on artificial intelligence according to claim 2, characterized in that: In step S203, the weight is determined according to the in-degree and out-degree of the computing task node, and the scheduling priority is determined according to the weight, specifically: The computing task nodes with fewer in-degrees and more out-degrees have greater weights and higher scheduling priorities. The calculation formula is W i =in i *0.2 + out i *0.8, where W i is the weight of computing task node i, in i is the in-degree of the computation task node i, out i is the out-degree of the computation task node i.

5. The method for visual scheduling of computing tasks based on artificial intelligence according to claim 2, characterized in that: In step S4, the computing task operation log is analyzed: the error log in the computing task is segmented, and the word frequency analysis is performed using the TF-IDF algorithm, and the error types are archived and classified for unified processing, which specifically includes: S401, performing log level filtering on the task running log; S402, performing corpus segmentation on the logs at the warning level and the error level; S403, using the TF-IDF algorithm to perform word frequency analysis on the warning level logs and the error level logs, and classify the error types.

Images