Data lake-based data management system and method

Abstract

The embodiment of the invention provides a data lake-based data management system and method, and relates to the technical field of big data. The system comprises a plurality of computing engines, a plurality of data sources and a virtual engine device, each computing engine is used for sending a data access request and identification information of a to-be-accessed data source to the virtual engine device and receiving a response result which is fed back by the virtual engine device and corresponds to the data access request; the virtual engine device is used for sending a data access requestto the data source with the identification information; and feeding back a response result fed back by the data source with the identification information to the computing engine sending the data access request, and each data source is used for determining a response result corresponding to the data access request and feeding back the determined response result to the virtual engine device. Compared with the prior art, by applying the system provided by the embodiment of the invention, the repeated development work in the construction process of the data management system based on the data lake can be reduced.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data management system and method based on a data lake. Background technique [0002] Currently, data lakes are increasingly being used in various fields, such as logistics, manufacturing, project management, business management etc. [0003] In related technologies, when using a data lake-based data management system for data management, the computing engine in the management system sends a data access request to the data source to be accessed through a pre-configured access interface that matches the data source to be accessed, Furthermore, the data source to be accessed receives the data access request and feeds back the response result corresponding to the data access request to the computing engine. [0004] Obviously, since different business applications use different computing engines, and each computing engine requires different access interfaces to access d...

AI Technical Summary

Problems solved by technology

[0004] Obviously, since different business applications use different computing engines, and each computing engine requires different access interfaces to access different data sources, then, when using the above-mentioned related technologies, if there are multiple computing engines and multiple data sources In the construction process of the data management system, there will be a lot of access interface development work, which makes the construction process more cumbersome

Images