System and method for internet big data task scheduling based on life cycle model

A life cycle and task scheduling technology, applied in data processing applications, instruments, office automation, etc., can solve the problems of lack of scheduling logic, error-prone, difficult to use, etc., and achieve the effect of reasonable cluster resources, high degree of automation, and reasonable utilization

Active Publication Date: 2016-06-29
上海晶赞企业管理咨询有限公司
View PDF5 Cites 13 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0010] The above-mentioned data task scheduling system currently has the following problems: (1) It is difficult to use, and it is difficult for users to directly and effectively interact with the system; (2) Lack of strict scheduling logic, although all task dependencies are managed through a direct...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • System and method for internet big data task scheduling based on life cycle model
  • System and method for internet big data task scheduling based on life cycle model
  • System and method for internet big data task scheduling based on life cycle model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] The specific implementation manners of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0050] The first purpose of the present invention is to build a system of Internet big data task scheduling based on life cycle model, please refer to figure 1 ,include:

[0051] A. First, establish a data task life cycle model.

[0052] See figure 2 , in the entire life cycle of a data task, there are three types of personnel involved: demand personnel, developers, and operation and maintenance personnel.

[0053] The data task lifecycle consists of four phases:

[0054] Data requirement stage: The requirement personnel put forward the data requirement.

[0055] Data development phase: Developers complete the design of data tasks.

[0056] Data execution stage: operation and maintenance personnel complete the online, execution and monitoring of data tasks.

[0057]Data execution result stage: Operation and maintenance pe...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

Provided is a system and method for internet big data task scheduling based on a life cycle model.The task scheduling system is constructed by designing a task expression method based on a data task life cycle model by establishing the data task life cycle model.The task scheduling system comprises an interface layer, a storage layer, a metadata layer and an execution layer.The metadata layer abstracts task instances and the dependencies of the task instances into an attributed graph, nodes in the attributed graph represent the task instances, node attributes include parameters of the task instances, sides in the attributed graph represent the dependencies of the task instances, and the task instances are scheduled through the attributed graph.The system can automatically derive the dependencies of tasks and is higher in automation degree and reliability.After the completion of data task development, task execution can be controlled only by submitting different instantiated parameters, the efficiency is higher, and the system is more intelligent.

Description

technical field [0001] The invention relates to the technical field of data service processing, in particular to a system and method for scheduling Internet big data tasks based on a life cycle model. Background technique [0002] Big data technology is a field that has developed extremely rapidly in recent years, and it is an important cornerstone to support mainstream Internet businesses such as modern Internet advertising, e-commerce, and 020. Take the Internet advertising business as an example. From 2011 to 2014 alone, the market size of Internet advertising has surpassed the size of newspaper advertising, ranking second, and the market size has continued to maintain rapid growth. The continuous fiery growth of Internet business continues to promote the development of big data technology. [0003] At present, the mainstream big data solution is distributed storage based on Hadoop cluster HDFS plus distributed computing engines such as MapReduce and Spark. Big data pro...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06Q10/10
CPCG06Q10/103
Inventor 汤奇峰侯杰
Owner 上海晶赞企业管理咨询有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap