Data lakehouse system with multi-type data support and event-driven governance and method thereof

Abstract

A data lake warehouse system and method with multi-type data support and event-driven governance connects data source devices, edge servers, and cloud servers. The edge server first receives multi-type raw data from the data source devices in real time and performs preliminary cleaning to generate pre-processed data. Then, the cloud server automatically updates the artificial intelligence model and generates event responses. At the same time, detailed permissions are set and a data lineage graph is established so that when an abnormal event response is detected, data is traced according to the data lineage graph. This achieves the technical effects of improving data processing efficiency and governance security.

Claims

1. A data lake repository system with multi-type data support and event-driven governance, the system comprising: an edge server connected to at least one data source device, the edge server receiving raw data from the data source device in real time, the raw data comprising one or any combination of structured data, unstructured data, and event-driven data, and performing preliminary cleaning processing on the raw data to generate preprocessed data for transmission; and a cloud server connected to the edge server for receiving the preprocessed data, the cloud server comprising: a storage module for pre-storing an artificial intelligence model and storing the received preprocessed data; and a training module connected to the storage module, the training module comprising a model training engine and a machine learning operation (MLO) module. The model training engine (MLOps) is used to automatically update the AI ​​model and generate a corresponding event response based on the preprocessed data, and transmit the updated AI model and the generated event response to the edge server through the machine learning operation; and a data governance module is connected to the training module. The data governance module sets a fine-grained permission based on the generated event response, and establishes a data lineage graph of the preprocessed data. When an abnormal event response is detected during the transmission of data from the cloud server to the edge server, the fine-grained permission is adjusted and data is tracked through the data lineage graph.

2. The data lake repository system with multi-type data support and event-driven governance as described in Request 1, wherein the event-type data is one of IoT events, webhook events, and log events, or any combination thereof.

3. A data lake repository system with multi-type data support and event-driven governance as described in Request 1, wherein the data governance module includes a graphics processing unit that parses transformation records, ETL (Extract, Transform, Load) scripts and model training records related to the preprocessed data, and establishes node-to-data tables, field-to-edge transformations and dependencies to form the data lineage graph.

4. A data lake repository system with multi-type data support and event-driven governance as requested in Item 1, wherein the data governance module includes an access control unit, which sets detailed permissions at the field level based on role-based access control (RBAC) to perform access management including read and write controls.

5. A data lake repository system with multi-type data support and event-driven governance as described in Request 1, wherein the data governance module includes a data lineage unit, which records the data lineage graph including data sources, transformation steps and event paths. When an abnormal event response is detected, an alarm is triggered through an audit log interface and a visual tracking report is automatically generated to complete the tracking control.

6. A data lake warehouse method with multi-type data support and event-driven governance, comprising the following steps: A method connects an edge server and a cloud server, wherein the edge server is connected to at least one data source device, and the cloud server pre-stores an artificial intelligence model; the edge server receives raw data from the data source device in real time, the raw data comprising one or any combination of structured data, unstructured data, and event-based data, and performs preliminary cleaning processing on the raw data to generate pre-processed data, and transmits the pre-processed data to the cloud server; the cloud server stores the received pre-processed data, and automatically updates the artificial intelligence model and generates a corresponding event response based on the pre-processed data through a model training engine, and transmits the updated artificial intelligence model and the generated event response to the edge server through Machine Learning Operations (MLOps); and the cloud server sets fine-grained permissions based on the generated event response, and establishes a data lineage graph of the pre-processed data. The graph, and when an abnormal event response is detected during the transmission of data from the cloud server to the edge server, triggers the adjustment of granular permissions and data tracking through the data lineage graph.

7. The data lake warehouse method with multi-type data support and event-driven governance as described in Request 6, wherein the event-type data is one of IoT events, webhook events, and log events, or any combination thereof.

8. The data lake warehouse method with multi-type data support and event-driven governance as described in Request 6, wherein the cloud server parses the transformation records, ETL (Extract, Transform, Load) scripts and model training records related to the preprocessed data, and establishes node-to-data tables, field-to-edge transformations and dependencies to form the data lineage graph.

9. The data lake repository method with multi-type data support and event-driven governance as described in request item 6, wherein the cloud server sets the detailed permissions at the field level based on role-based access control (RBAC) for access management including read and write control.

10. The data lake warehouse method with multi-type data support and event-driven governance as described in Request 6, wherein the cloud server records the data lineage graph including data sources, transformation steps and event paths, and when an abnormal event response is detected, an alarm is triggered through an audit log interface and a visual tracking report is automatically generated to complete the tracking control.

Images