Data processing task relationship setting method and system

Abstract

The invention relates to a data processing task relation setting method and system. The method includes the steps of obtaining at least one SQL script in a data processing task, carrying out morphology analysis and semantic analysis on SQL sentences in each SQL script in the at least one SQL script to build a data lineage relation of the SQL sentences, building a data lineage relation of the SQL scripts according to the data lineage relation of the SQL sentences, building a data lineage relation of the data processing task according to the data lineage relation of the SQL scripts in the at least one SQL script, determining data input and output of a data level and a task level of the data processing task, and determining and setting the relation between the data processing task and another data processing task according to the data lineage relation and the data level of the data processing task. Intelligent analysis and setting of the relation of the SQL data processing tasks can be achieved, the automation degree of data task scheduling configuration is improved and accuracy and efficiency of data operation and maintenance are achieved.

Description

technical field [0001] The invention relates to cloud computing technology, in particular to a method and system for setting data processing task relationships. Background technique [0002] In the big data environment of the cloud computing era, data is growing rapidly and expanding, and the number of various data processing tasks is also showing a trend of rapid growth. Information processing technology puts more emphasis on the ability to quickly obtain valuable information from massive data, which puts forward higher requirements for the efficient scheduling and execution of data processing tasks. [0003] Data processing tasks include a variety of data conversion-summarization and other processing. There are certain relationships between tasks (including dependencies and mutual exclusion relationships). Accurate task relationships are an important basis for efficient data scheduling. Taking the data warehouse system as an example, at present, in the scheduling of data ...

AI Technical Summary

Problems solved by technology

This requires the operation and maintenance personnel to have a good understanding of the whole data processing before and after, otherwise the task dependencies will be wrong, which will directly affect the results of data processing

But in fact, comprehensive operation and maintenance personnel do not exist, so there is an urgent need for a solution to overcome the defects of manually setting task relationships to improve the accuracy and efficiency of data operation and maintenance

Images