Data analysis job dependency generation method and system

Abstract

A method for generating data analysis job dependencies, comprising obtaining a job generation instruction containing data processing logic parameters to process data in a source database, generating a job according to the job generation instruction; obtaining basic information of the source database according to the job generation instruction ; Determine the parent node basic information and parent node attributes that the job depends on according to the source database basic information; store the job basic information of the job, the parent node basic information and the parent node attributes correspondingly , to generate job dependency mapping information of the job. The present application also provides a data analysis operation dependency generation system. Through the complete process of data conversion corresponding to a data processing logic parameter for each job dependency, the automatic generation and update of job dependencies is realized, which improves generation efficiency, reduces costs and has high accuracy.

Description

technical field [0001] The invention relates to the field of data analysis, in particular to a method and system for generating dependency relationships of data analysis operations. Background technique [0002] With the rapid development of Internet technology and the advent of the cloud era, big data analysis capabilities have gradually become one of the core competitiveness of enterprises. An efficient big data analysis architecture can help enterprises allocate resources faster and better, thus bringing huge advantages to enterprises. [0003] At present, the data analysis architecture relies on the Hadoop cluster for data storage and calculation at the bottom layer, data warehouse management at the middle layer based on Hive, and the upper layer provides users with a data analysis job submission interface, and each data analysis job is submitted through the submission interface. Among them, for the big data analysis architecture, it is not only necessary to consider th...

AI Technical Summary

Problems solved by technology

[0005] 1. Different data analysis jobs may be written and submitted by different data analysts. The parent job that a data analysis engineer's data analysis job depends on may be submitted by other data analysis engineers. Therefore, when multiple data analysis jobs are involved in dependency configuration In this case, it is necessary to obtain the information of the parent job with the corresponding data analysis engineer through offline communication, etc., which takes a lot of time, low efficiency, and high cost of configuration of dependencies

[0006] 2. The dependencies of data analysis jobs directly determine the scheduling order of jobs. A large-scale data analysis requirement is often completed by dozens or even hundreds of data analysis jobs. The dependencies between data analysis jobs are extremely complex. Dependencies are manually configured and maintained, which is not only costly, but also has a high probability of error. Once a dependency is wrong, it will lead to a wrong scheduling sequence, resulting in completely wrong analysis results

[0007] In related technologies, there is no effective solution for the above problems

Images