Data governance method based on double message queues

Abstract

The invention discloses a data governance method based on a double message queue, includes the following steps: 1) inputting data from a user data source through a data access tool, 2) storing the accessed data resources in a message queue, 3) extracting data from that message queue for preprocessing operation such as cleaning, 4) re-storing the pre-processed data into a message queue, 5) variousdata governance program respectively extract data from that message queue for governance, and then store the governance result into the message queue again; 6) the last governance program extracting data from the message queue, and after the governance is completed, the governance result being stored into a result database for use by subsequent processes. The method inserts the message queue before and after the data governance, buffers the data before and after the governance, realizes the streaming processing of the data, and optimizes the whole data processing link from the aspects of reliability, availability, scalability, data security and performance.

Description

technical field [0001] The invention belongs to the technical field of distributed computing and data processing, and in particular relates to a data management method based on dual message queues. Background technique [0002] Data governance is the process of reading data from one storage medium, going through a series of data governance links, and then storing it in another storage medium. For data governance with a large amount of data, there are two traditional methods: one is to read sequentially through a single thread, and then write to the target storage medium sequentially; the other is to read data in parallel through some rules and then write in parallel into the target stored procedure. However, in the process of governance, there will always be some problems: [0003] 1. There is a delay in data governance: the above two methods, either batch reading or writing, or timing reading and writing, cannot achieve real-time reading and writing, and are not applicabl...

AI Technical Summary

Problems solved by technology

[0011] In order to solve the problems of delay, untraceability, and low security in the existing data governance methods, the present invention provides a data governance method based on dual message queues. The method is inserted into the message queue before and after data governance , realize the buffering of the data before and after governance, realize the stream processing of data, and optimize the data processing link in terms of reliability, availability, scalability, data security and performance

Images