Collaboration method and device for data processing process

Abstract

The invention provides a collaboration method and device for a data processing process. The method comprises the following steps: building the configuration information of the relationship between a data processing procedure and the input / output of metadata, wherein the metadata comprises a data source and a data processing result; configuring the dispatching and panning information of each data processing process; and executing the corresponding data processing process according to the configuration information and the dispatching and panning information to realize the collaborating process among the data processing processes. According to the invention, the redundant execution waiting time is reduced, the data processing efficiency of an ODS (operational data store) system is improved, and the configuration of the data processing processes is simplified.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data processing process coordination method and device. Background technique [0002] The ODS system is a sharing platform for cross-system operation data, which undertakes the operation environment and analysis environment. ODS cleans, filters and integrates the data generated by the application system, and at the same time provides cross-system sharing data services such as near real-time operation reports for each application system. [0003] The data structure of the ODS system is as follows: figure 1 As shown, the data processing process includes: extracting business data from other business systems to the interface layer; cleaning, converting and loading data at the interface layer to the integration layer; applying and analyzing cross-system data with data from the integration layer to form Summarize the data of the layer and share the data of the aggre...

AI Technical Summary

Problems solved by technology

Therefore, the total execution time is too long and the data processing time is wasted.

[0006]In addition, whether it is time-based or data-processing-dependent, it is very cumbersome in actual use

Due to the large number of interfaces in the ODS system and the large order of magnitude of the data processing process, the process of configuration execution dependency and sequential execution sequence is quite complicated, and the business requirements for operators are quite high. Generally, only experienced maintenance personnel can understand most of the data processing process. It will be a very complicated job to require it to establish the dependencies of different data processing processes correctly, and the possibility of omissions in the configuration process is quite high

Images