Unlock instant, AI-driven research and patent intelligence for your innovation.

Big data operation task scheduling and monitoring system and method

A task scheduling and task technology, applied in the field of artificial intelligence, can solve problems such as task execution failure, task execution failure not restarting in time, and prolonging the time for obtaining calculation results, so as to improve the completion rate and accuracy rate, reduce server resource waste, and improve The effect of resource utilization efficiency

Active Publication Date: 2021-09-07
TERMINUSBEIJING TECH CO LTD
View PDF7 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] 1. It often takes up a lot of server resources. During the calculation process, the running time is often too long due to insufficient server resources. In severe cases, it will cause the task to overtime and fail.
[0004] 2. There are often big data computing tasks to be performed that depend on upstream data. If the upstream data is incomplete, the entire big data computing task will be incomplete or inaccurate, and all calculations need to be recalculated.
This not only wastes computing resources, but also prolongs the time to obtain computing results.
And even the wrong data will be used in the follow-up due to the untimely inspection of the data receiving personnel, causing serious consequences
[0005] 3. There are often big data calculation tasks to be performed that depend on certain interfaces. Due to different interfaces or problems with the data provided by the interfaces, the task execution fails or the calculation results are wrong.
[0006] 4. From time to time, the person in charge fails to discover the task execution failure or fails to restart the task in time, which will lead to the inability to obtain accurate calculation results in time, thus affecting the follow-up work

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data operation task scheduling and monitoring system and method
  • Big data operation task scheduling and monitoring system and method
  • Big data operation task scheduling and monitoring system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0120] This embodiment provides a big data computing task scheduling and monitoring system, such as figure 1 As shown, it includes: task interface module, task management module, task data scheduling engine, task required data AI evaluation module, task optimal execution AI evaluation module, task execution server module and task monitoring module;

[0121] The task interface module is used to send a defined task template to the upper-level system and receive a single task containing the task template information or a complex task composed of two or more tasks sent by the upper-level system after converting each task according to the task template. The task is output, and the template includes the task number, the data source required by the task, the task priority level, the pre-task dependency number, and the post-task dependency number; the data source required by the task is the corresponding data source obtained through the connection required to execute the task; the task...

Embodiment 2

[0180] This embodiment provides a big data computing task scheduling and monitoring method, such as Figure 5 shown, including the following steps:

[0181] S1. The task interface module sends a defined task template to the upper-level system and receives a single task containing the task template information or a complex task composed of two or more tasks sent by the upper-level system after converting each task according to the task template And output, the template includes the task number, the data source required by the task, the task priority level, the pre-task dependency number, and the task post-dependency number;

[0182] S2. The task management module receives the single task or the complex task output by the task interface module, and establishes a task schedule according to the task number, task priority, pre-task dependency number and post-task dependency number;

[0183] S3. The task execution server module generates and outputs a task processing request accord...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a big data operation task scheduling and monitoring system and method, and relates to the technical field of artificial intelligence, the system comprises a task data scheduling engine, a task required data AI evaluation module, a task optimal execution AI evaluation module and the like; the task required data AI evaluation module predicts and estimates the latest data volume based on the data of the data source required by the currently scheduled task by using the prediction model to obtain a data evaluation result; the task optimal execution AI evaluation module performs optimal operation evaluation on the task execution server module executing the currently scheduled task by using a sparse auto-encoder based on occupied resource data and available resource data of the task execution server module and resource data required to be occupied for executing the task to obtain an operation evaluation result; and the task data scheduling engine issues a task according to the data evaluation result and the operation evaluation result. The method has the advantages of high operation efficiency, high completion rate and high accuracy.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence, in particular to a system and method for dispatching and monitoring tasks of big data computing. Background technique [0002] In the era of big data, the generated data is growing explosively, so big data technology emerges as the times require, and resource scheduling is facing huge challenges. There are many instances that need to be scheduled every day, reaching tens of billions. Scheduling is a very difficult problem that needs to be balanced in all aspects. For example, fairness and real-time performance will be sacrificed when improving usage efficiency. In order to achieve the best scheduling effect, it is necessary to analyze resources and requirements such as CPU, Disk, and Memory, and to collect relevant information in real time. Executing big data computing tasks based on resource scheduling is a commonly used data processing method at present, but the execution of bi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/48G06F11/30G06F16/2458
CPCG06F9/4881G06F11/302G06F16/2462G06F16/2465
Inventor 边同昭廖旻可李玉敏方红波
Owner TERMINUSBEIJING TECH CO LTD