Big data platform workflow correctness verification method and system

Abstract

The invention provides a big data platform workflow correctness verification method and system, and the method comprises the steps: obtaining a workflow XML code file, and extracting the scheduling information and parameter information of a workflow; generating a first workflow file according to the scheduling information and the parameter information of the workflow; obtaining a scheduling scriptcode, and extracting a data table on which a script depends; generating a second workflow file according to the data table; judging the correctness of the workflow file by comparing the differences between the first workflow file and the workflow configuration file and between the second workflow file and the workflow configuration file, and generating a workflow correctness verification result.According to the method, the correctness of workflow configuration is automatically verified through two dimensions of developed workflow XML codes and scheduling scripts developed by developers, theproblem of workflow configuration is found in advance, it is guaranteed that workflow configuration conforms to code logic, adverse effects caused by human errors are prevented, and a powerful guarantee is provided for batch normal operation.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a method and system for verifying the correctness of a big data platform workflow. Background technique [0002] At present, the existing big data suite workflow development function only supports manual drag-and-drop development on the visualization page, and does not have the batch development workflow function. Most of the code (hive sql, shell, etc.) pre-depends on each data table in the platform. In the actual development process, developers generally apply for workflows by filling in documents, and fill in workflow names, steps, pre-dependent table names, and script names. [0003] In the existing big data platform scheduling system, the system will automatically trigger the execution of tasks in the workflow according to the workflow configuration of each tenant. The execution process generally checks whether the data files used in the code already exist, and ...

AI Technical Summary

Problems solved by technology

However, developers often fill in the application form incorrectly due to code modification or other reasons. As a result, developers often omit or incorrectly fill in the prefix (data table partition) and data table name that each task instance depends on in the workflow configuration of the developer's application, which leads to batch operation. Batch exceptions occur due to non-existence of data or incorrect filling in the process

When the person in charge of the workflow develops the workflow according to the workflow configuration file of the requisition form, the errors of the developers will be inherited, and the developers themselves will also make mistakes in filling in the data table name and missing data table dependencies when developing the workflow, and then lead to batch error

Misconfiguration of the workflow may lead to data processing errors, which in turn affect the use of normal functions and adversely affect the development of normal business functions of the bank

[0004] To sum up, in the existing big data suite, the workflow adopts a method similar to a flow chart to set each configuration item. Due to the inconvenient interface, there are often negligence and omissions in the manual configuration of the workflow, which cannot fully reflect the integrity of the workflow tasks. , correctness, which may lead to batch errors and other important effects

[0005] Therefore, there is an urgent need for a technical solution that can overcome the above problems and avoid the impact of abnormal batches caused by human workflow configuration errors on the big data platform

Images