System and method for internet big data task scheduling based on life cycle model

Abstract

Provided is a system and method for internet big data task scheduling based on a life cycle model.The task scheduling system is constructed by designing a task expression method based on a data task life cycle model by establishing the data task life cycle model.The task scheduling system comprises an interface layer, a storage layer, a metadata layer and an execution layer.The metadata layer abstracts task instances and the dependencies of the task instances into an attributed graph, nodes in the attributed graph represent the task instances, node attributes include parameters of the task instances, sides in the attributed graph represent the dependencies of the task instances, and the task instances are scheduled through the attributed graph.The system can automatically derive the dependencies of tasks and is higher in automation degree and reliability.After the completion of data task development, task execution can be controlled only by submitting different instantiated parameters, the efficiency is higher, and the system is more intelligent.

Description

technical field [0001] The invention relates to the technical field of data service processing, in particular to a system and method for scheduling Internet big data tasks based on a life cycle model. Background technique [0002] Big data technology is a field that has developed extremely rapidly in recent years, and it is an important cornerstone to support mainstream Internet businesses such as modern Internet advertising, e-commerce, and 020. Take the Internet advertising business as an example. From 2011 to 2014 alone, the market size of Internet advertising has surpassed the size of newspaper advertising, ranking second, and the market size has continued to maintain rapid growth. The continuous fiery growth of Internet business continues to promote the development of big data technology. [0003] At present, the mainstream big data solution is distributed storage based on Hadoop cluster HDFS plus distributed computing engines such as MapReduce and Spark. Big data pro...

AI Technical Summary

Problems solved by technology

[0010] The above-mentioned data task scheduling system currently has the following problems: (1) It is difficult to use, and it is difficult for users to directly and effectively interact with the system; (2) Lack of strict scheduling logic, although all task dependencies are managed through a directed acyclic graph (DAG) , but in the actual scheduling job, it is difficult to track and restore the state on the DAG; (3) the current mainstream task scheduler is to manually define the DAG directly to schedule tasks

A major disadvantage of this is that the DAG definition process is complex and error-prone

Images