Data lake management system, method and device and electronic equipment

Abstract

The invention relates to the technical field of big data, in particular to a data lake management system, method and device and electronic equipment, and aims to improve the efficiency and flexibility of data lake management. The method comprises the steps that a query calculation engine management module applied to a data lake management system configures a specified query calculation engine in a data lake, and establishes a resident resource pool through the query calculation engine; after a library table structure change request or a data processing request for the data lake is received through the resident resource pool, starting at least one target resident session in the resident resource pool through a query calculation engine; the library table structure change request is triggered by the object through a metadata management interface provided by a metadata management module in the data lake management system, and the data processing request is triggered by the object through a big data processing interface provided by a big data processing module in the data lake management system; and processing the corresponding request through the at least one target resident session.

Description

Background Art

[0002] Data Lake is a highly scalable and cost-effective storage architecture designed to effectively manage and store data resources of various types and formats. Apache HUDI (Apache Hadoop UpsertsDeletes and Incrementals) is an open source data management and storage framework. It is widely used in data lake construction because of its support for incremental data processing and efficient data synchronization. HUDI can synchronize metadata with data warehouses such as Hadoop Integrated Virtual Environment (Hive), and allow users to perform offline queries through tools such as Hive, but does not support Hive-based table structure changes and data addition, deletion, modification, and update operations.

[0003] In related technologies, only the distributed data processing framework SPARK or the stream processing framework FLINK can be used to change the table structure of the HUDI data lake table and add, delete, modify, and update data on HUDI. The entire process is very lengthy. At the same time, there is a certain technical cost for development using SPARK or FLINK, and it is inevitable to use complex table creation statements, which requires users to master complex technologies and development processes, increasing the cost of use.

[0004] In summary, how to improve the efficiency and flexibility of data lake management is an urgent issue to be addressed. Summary of the invention

[0005] Embodiments of the present application provide a data lake management system, method, device, and electronic device to improve the efficiency and flexibility of data lake management.

[0006] A data lake management system provided in an embodiment of the present application includes: a metadata management module, a big data fast processing module, and a query computing engine management module;

[0007] The metadata management module is used to provide a metadata management interface, and send a request for changing the library and table structure of the data lake triggered by an object through the metadata management interface to the query computing engine management module;

[0008] The big data processing module is used to provide a big data processing interface, and send the data processing request of the data lake triggered by the object through the big data processing interface to the query calculation engine management module;

[0009] The query computing engine management module is used to configure a specified query computing engine in the data lake and establish a resident resource pool through the query computing engine; after receiving the library table structure change request or the data processing request through the resident resource pool, start at least one target resident session in the resident resource pool through the query computing engine; and process the corresponding request through the at least one target resident session.

[0010] A data lake management method provided in an embodiment of the present application is applied to a query computing engine management module in a data lake management system, including:

[0011] Configure a specified query computing engine in the data lake, and establish a resident resource pool through the query computing engine;

[0012] After receiving a request for changing the library table structure or a data processing request for the data lake through the resident resource pool, at least one target resident session in the resident resource pool is started through the query computing engine; the request for changing the library table structure is triggered by an object through a metadata management interface provided by a metadata management module in the data lake management system, and the data processing request is triggered by an object through a big data processing interface provided by a big data processing module in the data lake management system;

[0013] The corresponding request is processed through the at least one target resident session.

[0014] Optionally, the library table structure change request or data processing request is parsed through the at least one target resident session to obtain a structured query language SQL statement corresponding to the request processing;

[0015] The structured query language is used to call the computing instance in the query computing engine to process the corresponding request.

[0016] The request is parsed into a structured query language SQL statement in the above way, and then the target resident session passes the parsed SQL statement to the query computing engine, which executes the SQL statement through its internal computing instance. The query computing engine calls the computing instance (such as Spark, Flink, etc.) to execute SQL queries or database structure change tasks, thereby completing data processing or structure modification tasks.

[0017] Optionally, for each target resident session, perform the following operations:

[0018] Configure a preset survival check time for the target resident session, and periodically poll and run corresponding SQL statements according to the survival check time;

[0019] If the target resident session can return normally, it indicates that the target resident session runs normally;

[0020] If the target resident session fails to return normally, the target resident session is restarted through a restart mechanism.

[0021] Through the above methods, the stability and health of the target resident session are ensured. Regular survival checks can detect session anomalies in a timely manner to avoid idle system resources or inability to process query requests due to session failures, thereby improving system reliability. Automated fault recovery: When the target resident session is detected to be unavailable, the system can automatically trigger the restart mechanism to quickly restore the operation of the session. This reduces manual intervention and improves the system's automated management capabilities and operation and maintenance efficiency. Maintain high availability of the resource pool: By restarting abnormal sessions and restoring normal status, ensure that sessions in the resident resource pool are always available, thereby ensuring efficient response and execution of query requests and data processing tasks.

[0022] Optionally, a resource allocation strategy of the query computing engine is configured so that the query computing engine dynamically allocates computing resources according to a request of an object.

[0023] Through the above configuration, you can dynamically configure the allocation strategy of Spark computing resources to ensure that the use of computing resources is dynamically adjusted according to the load of different query tasks, thereby achieving dynamic expansion and contraction of resources.

[0024] Optionally, by configuring a preset number of resident sessions, a resident resource pool including multiple resident sessions is established to respond in real time to library table structure change requests or data processing requests for the data lake, and allocate target resident sessions according to the requests.

[0025] By presetting a certain number of resident sessions and keeping them in the system, it can be ensured that the system can handle multiple concurrent requests at any time and avoid waste of resources due to delays in session creation.

[0026] A data lake management method provided in an embodiment of the present application is applied to a big data processing module in a data lake management system, including:

[0027] Provides a big data processing interface;

[0028] A data processing request of the data lake triggered by the object through the big data processing interface is sent to a query computing engine management module in the data lake management system, so that after the query computing engine management module receives the data processing request through a resident resource pool, it starts at least one target resident session in the resident resource pool through a query computing engine, and processes it through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

[0029] Optionally, by configuring a big data processing cycle, the data lake management system periodically schedules and processes the big data in the data lake.

[0030] The big data processing cycle managed by the periodic scheduling of the above-mentioned big data processing tasks can enable the data lake management system to achieve automation and flexibility in periodic task processing, reduce the need for human intervention, and improve the efficiency of data management and updating.

[0031] A data lake management method provided in an embodiment of the present application is applied to a metadata management module in a data lake management system, including:

[0032] Provides metadata management interface;

[0033] A request for changing the library and table structure of the data lake triggered by the object through the metadata management interface is sent to the query computing engine management module, so that after the query computing engine management module receives the request for changing the library and table structure through the resident resource pool, it starts at least one target resident session in the resident resource pool through the query computing engine, and processes it through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

[0034] A data lake management device provided in an embodiment of the present application is applied to a query computing engine management module in a data lake management system, including:

[0035] A configuration unit, used to configure a specified query computing engine in the data lake, and establish a resident resource pool through the query computing engine;

[0036] A starting unit, configured to start at least one target resident session in the resident resource pool through the query computing engine after receiving a library table structure change request or a data processing request for the data lake through the resident resource pool; the library table structure change request is triggered by an object through a metadata management interface provided by a metadata management module in the data lake management system, and the data processing request is triggered by an object through a big data processing interface provided by a big data processing module in the data lake management system;

[0037] A processing unit is used to process the corresponding request through the at least one target resident session.

[0038] A data lake management device provided in an embodiment of the present application is applied to a big data processing module in a data lake management system, including:

[0039] A first interface providing unit, used to provide a big data processing interface;

[0040] The first sending unit is used to send a data processing request of the data lake triggered by the object through the big data processing interface to a query computing engine management module in the data lake management system, so that after the query computing engine management module receives the data processing request through a resident resource pool, it starts at least one target resident session in the resident resource pool through a query computing engine, and processes through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

[0041] A data lake management device provided in an embodiment of the present application is applied to a metadata management module in a data lake management system, including:

[0042] The second interface providing unit is used to provide a metadata management interface;

[0043] The second sending unit is used to send the library table structure change request of the data lake triggered by the object through the metadata management interface to the query computing engine management module, so that after the query computing engine management module receives the library table structure change request through the resident resource pool, it starts at least one target resident session in the resident resource pool through the query computing engine, and processes it through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

[0044] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned data lake management methods.

[0045] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform any step of the above-mentioned data lake management method.

[0046] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs any step of the above-mentioned data lake management method.

[0047] The beneficial effects of this application are as follows:

[0048] The embodiments of the present application provide a data lake management system, method, device and electronic device. Specifically, the data lake management system proposed in the present application includes: a metadata management module that provides a human-computer interaction interface, a big data fast processing module, and a query computing engine management module that is pre-configured with a specified query computing engine. The first two modules support objects to trigger various requests through the human-computer interaction interface, and the last module supports fast processing of these requests.

[0049] Specifically, the metadata management module and the big data processing module both provide a human-computer interaction interface, so that the object can directly initiate a request to change the library table structure or a data processing request through the interface. Through this human-computer interaction interface, the data lake is managed uniformly, reducing the cost of data lake management. Then the metadata management module and the big data processing module send the corresponding request to the query computing engine management module. The query computing engine management module first configures the specified query computing engine in the data lake and establishes a resident resource pool, so that the query computing engine is always on standby, avoiding the initialization overhead for each request. The resident resource pool can quickly respond to different types of queries and data processing requests by presetting multiple resident sessions when the system starts, thereby improving the query response speed and system processing efficiency. In this way, the system can provide sufficient computing resources during peak demand and avoid unnecessary computing overhead when resources are idle, thereby optimizing the efficiency of resource use.

[0050] Then, after receiving a request to change the database table structure or a data processing request, the query computing engine can quickly start the target resident session to process the corresponding request. In this way, the system can respond to requests involving metadata management or big data processing in the data lake in real time, avoiding the delay of recreating sessions or computing instances, thereby speeding up the processing of requests. This efficient resource scheduling mechanism enables the system to respond to different types of requests and reduces the time it takes for the data lake to process requests.

[0051] Other features and advantages of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0053] Figure 1 A system architecture diagram of a data lake management system provided in an embodiment of the present application;

[0054] Figure 2 A flowchart of an implementation method of a data lake management method provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of a data lake library table preview interface of a metadata management interface provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of creating a data lake library table for a metadata management interface provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of creating a data lake clone table / delete table interface of a metadata management interface provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of a data lake table structure design interface of a metadata management interface provided in an embodiment of the present application;

[0059] Figure 7 A flowchart of another data lake management method provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of a big data processing interface provided for an application embodiment;

[0061] Fig. 9 A schematic diagram of a big data processing task cycle scheduling management provided in an embodiment of the present application;

[0062] Fig.10 A flowchart of another data lake management method provided in an embodiment of the present application;

[0063] Fig.11 A schematic diagram of a processing flow of a resident resource pool provided in an embodiment of the present application;

[0064] Fig.12 An implementation flow chart of a data lake management system provided in an embodiment of the present application;

[0065] Fig.13 A schematic diagram of the structure of a data lake management device provided in an embodiment of the present application;

[0066] Fig.14 A schematic diagram of the structure of another data lake management device provided in an embodiment of the present application;

[0067] Fig.15 A schematic diagram of the structure of another data lake management device provided in an embodiment of the present application;

[0068] Fig.16A schematic diagram of an electronic device of a data lake management device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0070] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0071] See also Figure 1 As shown, it is a system architecture diagram of a data lake management system provided in an embodiment of the present application.

[0072] In the embodiment of the present application, the data lake management system 100 includes but is not limited to three parts: a metadata management module 101, a big data fast processing module 102, and a query computing engine management module 103.

[0073] First combine Figure 1 , briefly introduce the data lake management system of this application:

[0074] Metadata management module 101: used to provide a metadata management interface, and send the data lake table structure change request triggered by the object through the metadata management interface to the query calculation engine management module.

[0075] Among them, the metadata management module 101 supports the management of the data lake library and table metadata life cycle through the metadata management interface, including but not limited to previewing the library and table structure, table registration (creation), table cloning, table deletion, and table structure modification operations, and generates corresponding library and table structure change requests for processing by the query calculation engine management module 103.

[0076] The big data fast processing module 102 is used to provide a big data processing interface and send the data processing request of the data lake triggered by the object through the big data processing interface to the query calculation engine management module.

[0077] The big data fast processing module 102 provides an efficient big data processing development interface, supporting large-scale processing and analysis operations on data in the data lake. Objects can initiate data processing requests through this module, and pass the request to the query calculation engine management module 103 for corresponding processing through online operation.

[0078] Query computing engine management module 103: used to configure a specified query computing engine in the data lake, and establish a resident resource pool through the query computing engine; after receiving a library table structure change request or a data processing request through the resident resource pool, start at least one target resident session in the resident resource pool through the query computing engine; process the corresponding request through at least one target resident session

[0079] The query computing engine management module 103 is the core module of the system, which is responsible for configuring the specified query computing engine in the data lake and creating a resident resource pool.

[0080] In the embodiment of the present application, the resident resource pool is a collection of resident sessions that run continuously, which is specifically used to support the request processing of the query computing engine. The resource pool ensures that the query computing engine can respond quickly when receiving a request through pre-configured computing resources and resident sessions, thereby avoiding the delay problem of restarting computing resources for each request. The resident resource pool is always on standby by maintaining a certain number of resident idle sessions so that sudden requests for library table structure changes and data processing requests can be processed quickly.

[0081] The query computing engine management module receives the library table structure change request from the metadata management module 101 and / or the data processing request from the big data fast processing module 102. Through one or more sessions in the resident resource pool, the query computing engine management module can quickly respond to and process various requests, thereby improving the efficiency and flexibility of data lake management operations.

[0082] The above-mentioned data lake management system can be applied to data lakes with open source Hadoop technology as the core. Open source Hadoop technology is a complete set of technical solutions for the data source layer, data transmission layer, data storage layer, resource management layer, data computing layer, task scheduling layer, and business model layer.

[0083] The following describes the data lake management method provided by the exemplary embodiment of the present application in combination with the system architecture described above and with reference to the accompanying drawings.

[0084] See also Figure 2 As shown, it is a flowchart of an implementation of a data lake management method provided in an embodiment of the present application, which is applied to a metadata management module in a data lake management system, including the following steps S21-S22:

[0085] S21: Provides a metadata management interface.

[0086] In an embodiment of the present application, the metadata management interface is used to provide convenient library and table management functions, so that objects (referring to users, such as data developers) can easily operate library and tables in the data lake.

[0087] Considering that in actual applications, users may have different types of library table management requirements / library table structure change requests, the metadata management interface in this application includes but is not limited to the following categories:

[0088] (1) The data lake table preview interface is used to display the structure and content information of the existing tables in the data lake, so that users can view the status of the current tables and data overview.

[0089] See also Figure 3 As shown, it is a schematic diagram of a data lake library table preview interface of a metadata management interface provided in an embodiment of the present application.

[0090] At the top of the data lake table preview interface, users can select the engine type (such as HUDI) and engine instance (such as system_hudi) to filter and locate a specific data lake instance. The data lake table preview interface also provides a keyword search function, where users can enter keywords to quickly filter and find the required tables.

[0091] In the library table list area, the names of various libraries and tables are displayed, which makes it easier for users to view, select, and further operate the library table structure and content in the data lake.

[0092] (2) Data lake table creation interface: provides users with an operation entry to create new tables, and supports the definition of table structure and attribute configuration to meet data management requirements.

[0093] See also Figure 4 As shown, it is a schematic diagram of creating a data lake library table in a metadata management interface provided in an embodiment of the present application.

[0094] The data lake table creation interface can be accessed through Figure 3 The "Create" button in the data lake table preview interface is Figure 4 After the user clicks the "Create" button, the interface will display the options of creating a new database and a new table. The user can choose to create a new database or a new table, and further configure the table structure and properties, such as the table engine type, engine instance, database name, etc., which will be specifically introduced in the data lake table structure design interface.

[0095] (III) Data Lake Clone Table / Delete Table Interface: supports quick copying (cloning) and deletion of database tables, making it easier for users to perform data backup and database table management.

[0096] See also Figure 5 As shown, it is a schematic diagram of creating a data lake clone table / delete table interface of a metadata management interface provided in an embodiment of the present application.

[0097] The clone table and delete table interfaces of the data lake can be accessed through Figure 3 The specific table item in the data lake library table preview interface shown in the figure can be entered. Users can click the table name to choose to clone or delete the table. Through the cloning operation, users can quickly create a copy of the source table for data backup or experimental testing; through the deletion operation, users can remove tables that are no longer needed, improving the flexibility of data management.

[0098] (IV) Data Lake Table Structure Design Interface: provides intuitive table structure definition and modification functions, and supports the addition, modification, and deletion of table fields.

[0099] See also Figure 6 As shown, it is a schematic diagram of a data lake table structure design interface of a metadata management interface provided in an embodiment of the present application. Figure 4 After clicking on the "New Table" button, the user can enter Figure 6 The table structure design interface is shown. In this interface, users can further configure the table structure and properties, such as field name, data type, partition information, etc.

[0100] In the embodiment of the present application, the table structure design interface includes but is not limited to the following parts:

[0101] (1) Table information: including but not limited to basic information such as engine type, engine instance, database, table name, Chinese name, storage period (you can choose "permanent" or custom period), etc.

[0102] The engine type indicates the type of data processing engine used, such as HUDI, which determines the storage and processing methods of the table;

[0103] Among them, HUDI is an open source solution for data lakes. HUDI is the abbreviation of Hadoop Updates and Incrementals. It is a data lake solution developed and open sourced by Uber. HUDI is a streaming lake warehouse integrated platform that supports the update of massive data blocks. It ensures that operations performed on the timeline are atomic and guarantees transactional nature. HUDI provides two table modes, copy-on-write tables and merge-on-read tables; and also supports three query modes, including snapshot query, incremental query, and read-optimized query.

[0104] The engine instance represents a specific engine instance in the data lake management system, such as system_hudi. Users can manage different data resources by selecting different engine instances. The database represents the storage location of the table. Users can select the database to which the table belongs. The table name is the unique identifier of the table, which is used to distinguish different tables. The Chinese name is the Chinese descriptive name of the table, which facilitates understanding of the meaning of the table. The storage period is used to set the life cycle of the table data. You can select "permanent" or a custom storage period to control the data life cycle and storage strategy of the table.

[0105] (2) Physical model information: Physical model information includes, but is not limited to, table type, partition type, pre-merge fields, cleanup strategy, and number of retained commits, which are used to customize the storage mode and data update strategy of the table.

[0106] Among them, table types include Copy-on-Write (copy on write) and Merge-on-Read (merge on read) modes. The Copy-on-Write table type directly overwrites old data when writing data, which is suitable for scenarios with low update frequency and high query performance requirements; the Merge-on-Read table type merges changes when reading, which is suitable for scenarios with frequent data updates and high latency tolerance.

[0107] The partition type is used to set the partitioning method of the table. You can choose a partitioned table or a non-partitioned table. The partitioned table helps improve query performance, especially in scenarios with large data volumes. The pre-merge field is used to specify the basis field for the merge operation. Records will be merged based on this field when data is inserted or updated. The cleanup strategy is used to manage the use of storage space and the data life cycle. For example, it can be configured to clean up regularly according to data expiration time. The number of retained commits refers to the number of historical commit records retained to trace the change history or perform data recovery.

[0108] (3) Field information: Field information provides field operation functions and field detailed information configuration. Field operation functions include but are not limited to adding fields, introducing fields, batch deletion, moving fields up and down, etc., which facilitate users to flexibly adjust fields.

[0109] The detailed information of the field includes attributes such as field name, field Chinese name, field type, field length, required, and primary key. The field name is the unique identifier of the field; the field Chinese name is the Chinese description of the field, which is convenient for understanding the meaning of the field; the field type indicates the data type of the field, such as int, string, etc., which is used to define the storage type of the field; the field length is used to limit the storage length of the field; required is used to control whether the field is required to ensure data integrity; whether it is the primary key sets whether the field is used as the primary key, which is used to uniquely identify records in the table.

[0110] At the same time, users can add, delete, and modify field attributes through the field information interface to meet the needs of table structure adjustment.

[0111] Through the configuration of the above parts, the table structure design interface enables users to efficiently and intuitively complete the definition and management of the data lake table structure.

[0112] It should be noted that the several sub-interfaces of the metadata management interface listed above are just simple examples. In addition, other sub-interfaces related to the metadata management interface are also applicable to the embodiments of the present application and will not be described in detail here.

[0113] Through the above metadata management interface, the data lake library tables are uniformly managed through a graphical interface. Users do not need to master complex command line operations or distributed computing frameworks, which reduces the technical threshold and operation and maintenance costs, making the management of the data lake more intuitive and efficient. The system supports flexible library and table management operations. Users can flexibly and quickly perform data table operations through interfaces such as library and table preview, creation, cloning / deletion, and table structure design, meeting the efficient management needs of data lake resources in different business scenarios.

[0114] The table structure design interface provides intuitive field addition and deletion, field attribute modification, and partition strategy configuration functions, allowing users to easily adjust the table structure to adapt to changes in data storage requirements and improve the flexibility of data lake table structure management.

[0115] The above briefly lists several different types of metadata management interfaces, which are designed to support users' diverse library and table management needs and library and table structure change requests. In the process of data lake management, users often need to perform various operations on libraries and tables, such as creating new tables, cloning existing tables, deleting tables that are no longer needed, or modifying the structure and properties of tables. These operations not only involve the maintenance of basic information of libraries and tables, but may also include the configuration of advanced properties to meet specific data storage and computing needs.

[0116] To this end, the embodiment of the present application designs multiple metadata management interface modules, which meet the various practical needs of users in library and table management through an intuitive and layered interface layout. Each interface provides a clear operation entry and corresponding functions. For example, the library and table creation interface supports the selection of a new library or table, and allows users to configure the table structure in detail; the clone table interface allows users to quickly generate a copy of the table, which is convenient for data backup and testing; the delete table interface provides an operation process for safely deleting the table to prevent data loss caused by misoperation.

[0117] In addition, in the table structure design interface, users can further define the detailed structure of the table, such as field name, data type, partition information, etc., which provides flexible support for the refined management and customized configuration of the data lake. In short, the design of these metadata management interfaces can adapt to the library and table management needs in different scenarios, ensuring that users can efficiently and accurately operate and maintain the library and tables in the data lake, effectively improving the flexibility and operability of data lake management.

[0118] It should be noted that the above-mentioned types of metadata management interfaces are only simple examples, and this article does not make specific limitations on them. As user needs continue to change, other types of metadata management interfaces can be added or deleted, or the design of certain metadata management interfaces can be modified, etc., which will not be described one by one here.

[0119] S22: Send the request for changing the library and table structure of the data lake triggered by the object through the metadata management interface to the query computing engine management module, so that after the query computing engine management module receives the request for changing the library and table structure through the resident resource pool, it starts at least one target resident session in the resident resource pool through the query computing engine and processes it through the at least one target resident session. The resident resource pool is established by a specified query computing engine configured in the data lake.

[0120] In an embodiment of the present application, the above-mentioned request for changing the library and table structure of the data lake triggered based on the metadata management interface is sent to the query computing engine management module for processing by the query computing engine management module, wherein the query computing engine needs to start at least one target resident session in the resident resource pool. The resident resource pool and the target resident session will be specifically introduced in the query computing engine management module and will not be repeated here.

[0121] In the above implementation, a quick response to a request for changing the library table structure is achieved through a target session in a resident resource pool, thereby avoiding the high latency of reallocating resources for each request in the traditional method and improving the efficiency and real-time performance of data lake library table management.

[0122] After introducing the metadata management module in the data lake management system, the following will specifically introduce the big data fast processing module in the data lake management system.

[0123] See also Figure 7 As shown, it is a flowchart of another implementation of a data lake management method provided in an embodiment of the present application, which is applied to a big data fast processing module in a data lake management system, including the following steps S71-S72:

[0124] S71: Provides a big data processing interface.

[0125] In the embodiment of the present application, the big data processing interface provides a convenient data operation portal, allowing users to easily add, delete, modify, query, and change table structures of data in the data lake. The interface allows users to flexibly operate big data and table structures in the data lake through HUDI SQL statements.

[0126] In the above-mentioned metadata management module, table structure changes are mainly triggered through graphical interfaces and interactive operations. For example, users can select table items or click buttons through the graphical interface to perform specific operations without directly entering SQL statements, which is more friendly and intuitive for some users who are not familiar with SQL syntax. The table structure change operation of this module is relatively simple and is more suitable for some basic and commonly used table structure change requirements, such as adding fields, deleting tables, etc. In contrast, the big data fast processing module can not only handle data queries and processing requests, but also supports table structure change operations by directly entering SQL statements. It is suitable for users who need higher flexibility or have complex table structure management needs. The following will introduce it in detail.

[0127] See also Figure 8 As shown, it is a schematic diagram of a big data processing interface provided in an embodiment of the present application.

[0128] in, Figure 8 The interface contains multiple function buttons and SQL statement input area. The function buttons include "Save", "Run Online" and "Format". These buttons are used to save SQL operations, execute SQL statements through the Kyuubi JDBC interface to achieve online operation, and format SQL code to improve code readability.

[0129] Kyuubi is built on Apache Spark, providing a high-performance general-purpose JDBC and SQL execution engine, enabling users to process big data like ordinary data. It supports unified multi-tenant JDBC interface services and makes up for the lack of multi-tenancy, authorization, load balancing and high availability features of Spark Thrift Server. Spark is a distributed computing platform, a computing framework written in Scala, and a fast, general, and scalable big data analysis computing engine instance based on memory.

[0130] The following will use specific SQL statements as examples to illustrate the operation flow of the data processing interface:

[0131] (I) Modify existing data: The user can enter the HUDI SQL command for data operation in the SQL statement input area. In an optional implementation, the following operations can be performed:

[0132] (1) Insert data: INSERT INTO database name.table name SELECT field value 1, field value 2, ...; this statement inserts a new data row into the data lake table.

[0133] (2) Delete data: DELETE FROM database name.table name WHERE condition; this statement can delete data rows that meet the conditions.

[0134] (3) Update data: UPDATE database name. table name SET field name = 'field value' WHERE condition; this operation updates the specified field value of the data row that meets the conditions.

[0135] (ii) Table structure change: Users can adjust the table structure by entering corresponding SQL statements.

[0136] In an optional implementation, the following operations may be performed:

[0137] (1) Modify the table name: ALTER TABLE database name. old table name RENAME TO new table name; this operation renames the existing table.

[0138] (2) Add fields: ALTER TABLE library name. table name ADD COLUMNS (field name colType, field name colType1); used to add new fields to the table structure.

[0139] Modify field type: ALTER TABLE library name.table name CHANGE COLUMN old field name new field name colType; used to update the name or data type of the field.

[0140] (3) Modify table properties: ALTER TABLE library name. table name SET TBLPROPERTIES (key = 'value'); this statement is used to set or change the properties of a table.

[0141] pass Figure 8In the big data processing interface shown in the figure, users can flexibly perform various data operations and table structure adjustment operations, thereby achieving accurate processing of big data and efficient management of data lake table structure. Specifically, users can manually enter SQL statements to implement different operations according to specific needs, including data insertion, deletion, update, and table structure changes (such as renaming tables, adding fields, modifying field types, etc.). This free operation method is particularly suitable for scenarios that require high flexibility and is not limited to the preset options of the graphical interface. Compared with the relatively simple table structure change function in the metadata management module, the big data fast processing module supports more complex SQL operations, such as modifying table properties, conditional updates, batch data processing, etc. This function allows users to complete data operations and structure changes in a single interface, simplifying the operation process and suitable for users with complex data processing needs. The SQL statement input interface can be executed online with Kyuubi's JDBC interface, and users do not need to wait for the initialization of new sessions or computing resources, which significantly improves the response speed. Function buttons (such as "Save", "Run Online", and "Format") also further optimize the operation experience, allowing users to quickly write, execute, save, and view SQL code.

[0142] It should be noted here that the SQL statements for the big data processing interface listed above are just simple examples. In addition, other SQL statements for the big data processing interface are also applicable to the embodiments of the present application and will not be described one by one here.

[0143] Considering the lack of periodic scheduling in the development process for big data processing. During the development process of the data lake, due to the periodicity of the source data, periodic scheduling is required, and there is a lack of efficient scheduling methods. Therefore, this application configures the big data processing cycle so that the data lake management system periodically schedules and processes the big data in the data lake. Specifically, the big data processing tasks are periodically scheduled by configuring the workflow so that the data lake management system periodically processes the big data in the data lake.

[0144] Specifically, users can configure workflows to set task execution cycles, trigger conditions and other parameters to achieve automated task management, reduce manual intervention and improve data processing efficiency.

[0145] See also Fig. 9 As shown, it is a schematic diagram of a big data processing task periodic scheduling management provided in an embodiment of the present application.

[0146] The workflow configuration is used to configure the periodic scheduling management of big data processing tasks, including but not limited to the following parts:

[0147] (1) Basic attributes: used to display and define the basic information of the task. Including but not limited to name, process ID, person in charge, and description. Name refers to the name of the workflow, which is used to identify the task. Process ID refers to the number that uniquely identifies the process (for example, 9443) to facilitate system identification and management. Person in charge refers to the person in charge or administrator of the process (such as "admin"), who is responsible for the configuration and monitoring of the task. Description refers to providing descriptive information about the task to facilitate understanding of the purpose and operation details of the workflow. Process parameters are used to define global parameters to ensure that these parameters are consistently applied throughout the task execution process.

[0148] (2) Global parameters: Users can use the "plus sign" button to add the required global parameters so that they can be uniformly applied and called in tasks.

[0149] (3) Scheduling configuration: Control the execution mode and cycle of tasks so that tasks can be executed according to the scheduled time. The scheduling mode supports two modes: "manual" and "periodic". Manual means that the task needs to be manually triggered by the user, and periodic means that the task is automatically executed according to the set cycle;

[0150] If you select "Periodic" as the scheduling method, you can set the date range (including start date and end date) and specific execution time of the task by setting the start and end time and timing to realize the automatic execution of the scheduled task.

[0151] (4) Advanced configuration: Provides advanced settings for task execution to ensure the task's handling strategy in abnormal situations. Including failure strategy: Defines the operation mode of the task when it encounters a failure. The options include "Continue" or "End". Continue means that the task continues to execute subsequent tasks after failure, and End means that the task terminates the entire process when it fails to avoid potential error accumulation.

[0152] (5) Process priority: used to set the execution priority of the task. The default priority is "Medium". The system will prioritize high-priority tasks when resources are limited.

[0153] Through the workflow configuration of the periodic scheduling management of the above big data processing tasks, the data lake management system can achieve automation and flexibility in periodic task processing, reduce the need for human intervention, and improve the efficiency of data management and updating.

[0154] S72: Sending a data processing request of the data lake triggered by the object through the big data processing interface to a query computing engine management module in the data lake management system, so that after receiving the data processing request through the resident resource pool, the query computing engine management module starts at least one target resident session in the resident resource pool through the query computing engine, and processes through at least one target resident session. The resident resource pool is established by a specified query computing engine configured in the data lake.

[0155] In an embodiment of the present application, the data processing request of the data lake triggered by the above-mentioned big data processing interface is sent to the query computing engine management module for processing by the query computing engine management module. The query computing engine needs to start at least one target resident session in the resident resource pool, wherein the resident resource pool and the target resident session will be specifically introduced in the query computing engine management module and will not be repeated here.

[0156] In the above implementation, a quick response to data processing requests is achieved through the target session resident in the resource pool, avoiding the high latency of reallocating resources for each request in the traditional method, thereby improving the efficiency and real-time performance of data lake data processing.

[0157] Considering the high threshold of using metadata management modules and big data fast processing in related technologies, ApacheHudi can only use SPARK or FLINK technology to change the table structure of HUDI tables and add, delete, modify and update data in HUDI. There is a certain cost for users to use. This application proposes the above-mentioned metadata management interface and big data fast processing interface to simplify user operations, lower the threshold of use and improve user experience.

[0158] After introducing the metadata management module and big data fast processing module in the data lake management system, the query computing engine management module in the data lake management system will be specifically introduced below.

[0159] See also Fig.10 As shown, it is an implementation flowchart of another data lake management method provided in an embodiment of the present application, which is applied to the query computing engine management module in the data lake management system, including the following steps S101 to S103:

[0160] S101: Configure a specified query computing engine in the data lake, and establish a resident resource pool through the query computing engine.

[0161] The data lake management system implements data query and computing operations by configuring a specified query computing engine, and establishes a resident resource pool through the engine to handle various requests. The query computing engine refers to the core component used to perform SQL queries, data analysis, and computing tasks in the data lake. The resident resource pool refers to a resource pool that is pre-created and maintained based on the query computing engine, which contains multiple resident sessions for rapid response and processing of library and table management and data processing requests in the data lake management system, effectively reducing resource startup overhead and improving system response speed.

[0162] In an optional implementation, the query computing engine is Kyuubi, which acts as a query computing engine, supports SQL queries through its standardized JDBC interface, and provides a unified access portal to the upper layer. Kyuubi itself does not perform computing tasks, but manages and schedules backend computing instances, such as Spark, to complete actual data query and processing.

[0163] It should be noted here that the Kyuubi query computing engine is only an example, and the query computing engines applicable to this application are not limited to Kyuubi. Any query computing engine that can handle library table management and big data processing tasks can be used. The query computing engine can be a query computing engine that integrates computing instances (such as Spark, Flink, or Presto / Trino), or it can be a self-executing computing engine with library tables and big data processing capabilities. Therefore, this application does not specifically limit the choice of query computing engine.

[0164] The following uses the Kyuubi query computing engine integrated with the Spark computing instance as an example. Of course, the computing instance is not limited to Spark, but can also be other instances that support big data computing, such as Flink, Presto / Trino, etc. The specific choice depends on the needs and scenarios of the data lake management system.

[0165] First, you need to configure the query computing engine. Taking the Kyuubi query computing engine with integrated Spark computing instance as an example, the specific configuration is to configure the Kyuubi query computing engine with integrated Spark computing instance in the data lake Apache HUDI. Apache HUDI data lake is a data lake management solution specially designed for large-scale distributed data storage and processing. It combines the flexibility of the data lake and the efficient query capability of the database, allowing efficient incremental update, deletion, query and version management of data stored in the data lake.

[0166] In an optional implementation, the query calculation engine is configured with the following parameters, as shown below:

[0167] #hudi data reading

[0168] spark.sql.extensions

[0169] org.apache.spark.sql.hudi.HoodieSparkSessionExtension

[0170] #hudi serialization support

[0171] spark.serializer org.apache.spark.serializer.KryoSerializer

[0172] #huid support

[0173] spark.sql.catalog.spark_catalog

[0174] org.apache.spark.sql.hudi.catalog.HoodieCatalog

[0175] Through the above configuration, Kyuubi can be used to support the operation of Hudi data lake when starting Spark instance. Configuring Kyuubi query computing engine with integrated Spark computing instance in HUDI data lake can make full use of HUDI's powerful incremental data processing capabilities, version control capabilities and efficient storage format. At the same time, the distributed computing engine provided by Kyuubi can uniformly manage multiple computing instances to improve query performance and data efficiency.

[0176] Of course, the configuration methods listed above are just simple examples. In addition, other configuration methods are also applicable to the embodiments of the present application. In addition, other computing instances that support big data computing can also be configured in a similar manner, which will not be described one by one here.

[0177] The above introduces the configuration of the query computing engine Kyuubi in the HUDI data lake. Considering that the processing process of the data lake in the relevant technology is lengthy, there is a waste of resources and it is necessary to start SPARK or FLINK tasks for resource allocation, and the metadata structure changes are inconsistent with the data processing resource requirements. The applied resources cannot be evaluated in advance, which may lead to insufficient or excessive resource applications. This application will configure the resource allocation strategy of the query computing engine so that the query computing engine can dynamically allocate computing resources according to user requests.

[0178] Specifically, through dynamic configuration, the query computing engine can automatically allocate appropriate computing resources according to user requests, thereby achieving efficient resource utilization. In dynamic resource configuration, users can set some key parameters, such as the maximum number of containers, the minimum number of containers, the number of initialization containers, etc. Through these configurations, resource allocation can be flexibly adjusted according to the load changes of query tasks.

[0179] In the embodiment of the present application, the resource allocation strategy of the query computing engine can be implemented by dynamically configuring the resource allocation of the computing instance of the query computing engine (still taking Spark as an example). In an optional implementation, the parameter configuration of the dynamic allocation of computing resources of the specific computing instance is as follows:

[0180] #Dynamic resource allocation

[0181] spark.dynamicAllocation.enabled true

[0182] #Initialize the resource container to 1

[0183] spark.dynamicAllocation.initialExecutors 1

[0184] #The maximum resource container is 10

[0185] spark.dynamicAllocation.maxExecutors 10

[0186] #The minimum resource container is 1

[0187] spark.dynamicAllocation.minExecutors 1

[0188] In the above parameter configuration, first set spark.dynamicAllocation.enabled to true, which means that Spark is allowed to automatically increase or decrease the number of Executors according to the load. Then set the number of initialization resource containers (Executors) to 1, which means that when the Spark cluster starts, 1 Executor will be allocated by default. Next, define the maximum and minimum Executor numbers. The maximum number of Executors is set to 10, which means that Spark can start up to 10 Executors to process query tasks to avoid over-allocation of resources. The minimum number of Executors is 1, which means that even if the query load is light, Spark will retain at least 1 Executor to process tasks.

[0189] Through the above configuration, you can dynamically configure the allocation strategy of Spark computing resources to ensure that the use of computing resources is dynamically adjusted according to the load of different query tasks, and realize dynamic expansion and contraction of resources. For example, operations on metadata of data lake tables use fewer resources, while operations on HUDI data use more resources. This configuration automatically applies for resources based on user requests, and the initial value is relatively small, reducing resource waste.

[0190] The query computing engine and its resource allocation strategy are configured in the data lake. The following is a detailed description of how to establish a resident resource pool through the query computing engine:

[0191] By configuring a preset number of resident sessions, a resident resource pool containing multiple resident sessions is established to respond to database table structure change requests or data processing requests for the data lake in real time, and allocate target resident sessions based on the requests.

[0192] The resident resource pool is composed of a set of preset resident sessions, which are created and maintained in a resident state when the system is started. The main purpose of the resident session is to quickly respond to and load the data lake structure change requests (such as updating the library table structure, metadata operations) and data processing requests (such as batch data calculation, incremental query, etc.), reducing the data lake processing request time.

[0193] In the embodiment of the present application, in order to make the Kyuubi session resident and serve long-term operation, it is first necessary to set the session timeout to 0, so that the Kyuubi session will not time out and will continue to maintain the service status. After setting the timeout to 0, the Kyuubi session will continue to remain resident in the YARN cluster, which is suitable for library and table management operations, incremental queries, or any tasks that require long-term computing.

[0194] After setting the timeout to 0, Kyuubi also needs to set the session isolation level to ensure isolation and security of computing in a multi-user or multi-task environment. By properly configuring the isolation level, you can ensure that computing resources between different users or tasks do not conflict, improving system stability and security.

[0195] In an optional implementation, the session isolation level of Kyuubi can be configured to the USER level, which means that each user's query session is isolated to prevent resource competition between users. Different users will be assigned to different session pools to ensure independent operation. In addition to the USER level, Kyuubi also supports other isolation levels (such as SESSION, etc.), and the appropriate level can be selected according to actual needs. Selecting the USER level can ensure efficient resource isolation, so that each user will not be affected by other users' queries when executing queries. The session isolation level here is just a simple example, and this application does not specifically limit the specific session isolation level.

[0196] After setting the isolation level for the session, in order to ensure that there are enough resident sessions to handle multiple requests, you need to configure a preset number of resident sessions. You can specify the number of resident sessions through Kyuubi.engine.share.level.subdomain.

[0197] In an optional implementation, if the number of resident sessions is 3, it can be specified as follows:

[0198] Kyuubi.engine.share.level.subdomain=1

[0199] Kyuubi.engine.share.level.subdomain=2

[0200] Kyuubi.engine.share.level.subdomain=3

[0201] By presetting a certain number of resident sessions and keeping them in the system, it can be ensured that the system can handle multiple concurrent requests at any time and avoid waste of resources due to delays in session creation.

[0202] It should be noted here that configuring a specified query computing engine in the data lake and establishing a resident resource pool through the query computing engine is only performed once when the data lake management system is started, rather than requiring reconfiguration for each query request. The resident session pool ensures a fast response to query requests. By presetting a certain number of sessions, the overhead of session initialization is avoided, while improving resource utilization and the concurrent processing capability of the system. The system can flexibly respond to different requests for changes to the structure of different libraries or data processing requests during operation without repeating the configuration operation of the resource pool, maintaining an efficient and stable service status.

[0203] As mentioned above, when the data lake management system is started, a specified query computing engine is configured, and a resident resource pool containing resident sessions is established through the query computing engine. The following will specifically introduce the processing process of the resident resource pool for data lake library table structure change requests or data processing requests.

[0204] S102: After receiving a library table structure change request or a data processing request for the data lake through the resident resource pool, at least one target resident session in the resident resource pool is started through the query computing engine; the library table structure change request is triggered by the object through the metadata management interface provided by the metadata management module in the data lake management system, and the data processing request is triggered by the object through the big data processing interface provided by the big data processing module in the data lake management system.

[0205] In an embodiment of the present application, the resident resource pool is used to receive a library table structure change request and / or a data processing request for the data lake, and then start at least one target resident session in the resident resource pool.

[0206] In an optional implementation, the number of target resident sessions started is positively correlated with the number of requests. For example, when the system receives more requests for database table structure changes or data processing, more resident sessions are needed to process these requests simultaneously. In order to ensure the concurrent processing capability and response time of the system, the Kyuubi query calculation engine will dynamically select and start the target resident session according to the number of requests.

[0207] Assume that three data processing requests arrive at the same time, and the system needs to start three target resident sessions from the resident resource pool to process these requests respectively. If there are enough idle resident session resources in the pool, the system will directly allocate them; if the resident sessions in the pool are full, the system will start new sessions according to the load situation.

[0208] When the existing resident sessions in the resident resource pool cannot satisfy all requests, the system needs to expand the resource pool by dynamically adding resident sessions. This is to ensure that the system can adjust resources according to load requirements and handle concurrent requests. Kyuubi supports dynamic adjustment of the session pool size, and the system automatically starts new resident sessions based on the number of current requests and computing requirements.

[0209] When the load is high, Kyuubi will automatically create new sessions and add them to the resident session pool to handle additional requests in a timely manner.

[0210] After the request task is processed, if the resident session has not been used for a long time, the system can remove the idle session from the resource pool according to the preset policy to release computing resources. If the system needs the resident session again, it will restart the existing resident session resources.

[0211] Through the above process, after receiving a request for changing the library table structure or a data processing request for the data lake, at least one target resident session in the resident resource pool is started through the query computing engine. The following will explain how to process the request through the target resident session.

[0212] S103: Process the corresponding request through at least one target resident session.

[0213] In the embodiment of the present application, the received library table structure change request or data processing request is processed through the target resident session, specifically:

[0214] First, a library table structure change request or a data processing request is parsed through at least one target resident session to obtain a structured query language SQL statement for corresponding request processing;

[0215] Then, the structured query language is used to call the computing instance in the query computing engine to process the corresponding request.

[0216] After receiving a request to change the structure of a database table or a request to process data, the target resident session will first parse the request. Both requests to change the structure of a database table and data processing requests will be parsed into a SQL statement. These SQL statements are in a format that the query calculation engine can understand and execute.

[0217] In an optional implementation, the target resident session can first execute a query SQL statement, such as selectengine_id(), to obtain the applicationId corresponding to the current session, thereby identifying the task or job of the session and ensuring the correct association of the task. At this time, the hierarchical value (Kyuubi.engine.share.level.subdomain) of the session can also be stored in a database or metadata storage for subsequent query, resource management or session monitoring.

[0218] The request is parsed into a structured query language SQL statement in the above way, and then the target resident session passes the parsed SQL statement to the query computing engine, which executes the SQL statement through its internal computing instance. The query computing engine calls the computing instance (such as Spark, Flink, etc.) to execute SQL queries or database structure change tasks, thereby completing data processing or structure modification tasks.

[0219] In an optional implementation, the running status of the target resident session may also be checked by configuring a preset survival check time for the target resident session. The specific steps are as follows:

[0220] For each target resident session, perform the following operations:

[0221] Configure the preset survival check time for the target resident session, and periodically poll and run the corresponding SQL statement according to the survival check time;

[0222] The survival check time determines the check frequency, which is used to periodically execute the corresponding SQL query statements to verify the running status of the target resident session.

[0223] In an embodiment of the present application, according to the configured survival check time, SQL query statements are periodically polled and run, such as select engine_id(), which is used to check the status of the current session.

[0224] If the target resident session can return normally, it indicates that the target resident session is running normally;

[0225] If the target resident session fails to return normally, the target resident session is started again through the restart mechanism.

[0226] If the target resident session can return results normally, it indicates that the session is running normally. If the survival check finds that there is a problem with the target resident session, the system will trigger the restart mechanism. According to the hierarchical value (subdomain) of the session, the system can find the corresponding Yarn applicationId in the database and kill the corresponding session process by calling the Yarn API. After that, the system will restart the corresponding Kyuubi session to ensure that a healthy session remains in the resource pool for subsequent requests.

[0227] Among them, Yarn is a new Hadoop resource manager and a general resource management system that can provide unified resource management and scheduling for upper-level applications. Its introduction has brought huge benefits to the cluster in terms of utilization, unified resource management and data sharing.

[0228] Through the above method, the stability and health of the target resident session are ensured. Regular survival checks can detect session anomalies in a timely manner, avoid idle system resources or inability to process query requests due to session failures, and thus improve system reliability.

[0229] See also Fig.11 As shown, it is a schematic diagram of a processing flow of a resident resource pool provided in an embodiment of the present application.

[0230] Fig.11 It shows how the resident resource pool, with the cooperation of the query computing engine Kyuubi and the Yarn resource manager, handles client requests (i.e., library and table structure change requests or data processing requests triggered by users) and ensures high system availability and response efficiency.

[0231] Among them, Yarn, as a general Hadoop resource manager, provides unified resource management and scheduling, improving the resource utilization and data sharing efficiency of the cluster. Spark is a distributed computing platform, a computing framework written in Scala language, and a fast, general, and scalable big data computing instance based on memory. API is an application program interface (API: Application Program Interface). The application program interface is a set of definitions, programs, and protocols. The API interface enables mutual communication between computer software. In this application, the client API request interface connects the metadata management module with the query computing engine module or connects or connects the big data processing module with the query computing engine module.

[0232] Then, the resident resource pool receives the data lake table structure change request and the big data processing request, and the query calculation engine Kyuubi starts the target resident session in the resident resource pool ( Fig.11 The example in the figure shows 3), which are processed by Spark computing instances. At the same time, a regular survival check mechanism is configured for the target resident session. The session status is monitored by periodically executing SQL queries (such as selectengine_id()). If the session is found to be abnormal or unresponsive during the check, the system calls the YarnAPI to terminate the session process and automatically restart the session to ensure that the session in the resource pool is always available.

[0233] Finally, the query calculation engine module sends the structure processed by the data lake processing request to the data lake, completing the entire processing flow.

[0234] At the same time, the whole process is coordinated by the distributed coordination service cluster Zookeeper. ZooKeeper is a distributed distributed application coordination service. The whole process is coordinated by the distributed coordination service Zookeeper to ensure synchronization and consistency among multiple components.

[0235] Considering the characteristics of data lake based on Apache Hudi technology, metadata information can be synchronized with the Hive data warehouse after the table is created, and offline queries can be performed through Hive and PRESTO. However, it does not support Hive-based table structure changes and data addition, deletion, modification, and update operations.

[0236] Among them, Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism that can store, query and analyze large-scale data stored in Hadoop.

[0237] This application proposes that after the table is built for the Hudi data lake, the metadata information can be synchronized with the Hive data warehouse, and the library table structure change and data processing operations are performed through the three modules of the above-mentioned data lake management system. The specific operations of the HUDI data lake, Hive data warehouse and data lake management system are explained below.

[0238] See also Fig.12 As shown, it is an implementation flow chart of a data lake management system provided in an embodiment of the present application.

[0239] First, the new table is connected to the system through the table registration function of the data lake management system and synchronized to the Hive data warehouse. At this point, the table is registered and ready for subsequent data processing.

[0240] Secondly, the accessed data will enter the data lake system through the data lake entry tool, and the metadata of the data will be synchronized to the Hive data warehouse. Subsequently, the metadata management module of the data lake management system will manage this information in a unified manner to ensure that the table metadata in the data lake and Hive warehouse remain consistent.

[0241] Then, the data lake management system supports operations on the metadata of the data lake, including table creation, cloning, deletion, structural changes, etc. These operations are implemented through the metadata management module, and the changes are synchronized to the Hive data warehouse to ensure data consistency and system usability.

[0242] Then, the big data processing module of the data lake management system supports operations such as adding, deleting, modifying and querying data in the data lake. After completing the data processing, the system will resynchronize the table metadata in the data lake to ensure that the data in the data lake and the Hive warehouse are consistent.

[0243] at last, Fig.12 The overall demonstration shows how to operate and manage data in a data lake management system by integrating Apache Hudi and Hive data warehouse. The data lake based on Hudi in the prior art can support offline queries of Hive, but does not support table structure changes and data addition, deletion and modification operations based on Hive. The data lake management system of this application implements data management and synchronization through three core modules to ensure that the data in the data lake and data warehouse are consistent.

[0244] Based on the same inventive concept, the embodiment of the present application also provides a data lake management device, which is applied to the query computing engine management module in the data lake management system. Fig.13 As shown, it is a schematic diagram of the structure of the data lake management device 1300, which may include:

[0245] A configuration unit 1301 is used to configure a specified query computing engine in the data lake, and establish a resident resource pool through the query computing engine;

[0246] The starting unit 1302 is used to start at least one target resident session in the resident resource pool through the query computing engine after receiving a library table structure change request or a data processing request for the data lake through the resident resource pool; the library table structure change request is triggered by an object through a metadata management interface provided by a metadata management module in the data lake management system, and the data processing request is triggered by an object through a big data processing interface provided by a big data processing module in the data lake management system;

[0247] The processing unit 1303 is configured to process the corresponding request through the at least one target resident session.

[0248] Optionally, the processing unit 1303 is specifically configured to:

[0249] Parsing the library table structure change request or data processing request through the at least one target resident session to obtain a structured query language SQL statement corresponding to the request processing;

[0250] The structured query language is used to call the computing instance in the query computing engine to process the corresponding request.

[0251] Optionally, the processing unit 1303 is further configured to:

[0252] For each target resident session, perform the following operations:

[0253] Configure a preset survival check time for the target resident session, and periodically poll and run corresponding SQL statements according to the survival check time;

[0254] If the target resident session can return normally, it indicates that the target resident session runs normally;

[0255] If the target resident session fails to return normally, the target resident session is restarted through a restart mechanism.

[0256] Optionally, the configuration unit 1301 is further configured to:

[0257] The resource allocation strategy of the query computing engine is configured so that the query computing engine dynamically allocates computing resources according to the request of the object.

[0258] Optionally, the configuration unit 1301 is specifically configured to:

[0259] By configuring a preset number of resident sessions, a resident resource pool including multiple resident sessions is established to respond to library table structure change requests or data processing requests for the data lake in real time, and allocate target resident sessions according to the requests.

[0260] Based on the same inventive concept, the embodiment of the present application also provides another data lake management device, which is applied to a big data processing module in a data lake management system. Fig.14 As shown, it is a schematic diagram of the structure of the data lake management device 1400, which may include:

[0261] The first interface providing unit 1401 is used to provide a big data processing interface;

[0262] The first sending unit 1402 is used to send a data processing request of the data lake triggered by the object through the big data processing interface to a query computing engine management module in the data lake management system, so that after the query computing engine management module receives the data processing request through the resident resource pool, it starts at least one target resident session in the resident resource pool through the query computing engine, and processes it through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

[0263] Optionally, the first sending unit 1402 is further configured to:

[0264] By configuring the big data processing cycle, the data lake management system can periodically schedule and process the big data in the data lake.

[0265] Based on the same inventive concept, the embodiment of the present application also provides another data lake management device, which is applied to the metadata management module in the data lake management system. Fig.15 As shown, it is a schematic diagram of the structure of the data lake management device 1500, which may include:

[0266] The second interface providing unit 1501 is used to provide a metadata management interface;

[0267] The second sending unit 1502 is used to send the library table structure change request of the data lake triggered by the object through the metadata management interface to the query computing engine management module, so that after the query computing engine management module receives the library table structure change request through the resident resource pool, it starts at least one target resident session in the resident resource pool through the query computing engine, and processes it through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

[0268] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0269] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0270] After introducing the data lake management method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0271] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0272] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server. In this embodiment, the structure of the electronic device may be as follows: Fig.16 As shown, it includes a memory 1601 , a communication module 1603 and one or more processors 1602 .

[0273] The memory 1601 is used to store computer programs executed by the processor 1602. The memory 1601 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0274] The memory 1601 may be a volatile memory, such as a random-access memory (RAM); the memory 1601 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1601 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1601 may be a combination of the above memories.

[0275] The processor 1602 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1602 is configured to implement the above data lake management method when calling the computer program stored in the memory 1601 .

[0276] The communication module 1603 is used to communicate with terminal devices and other servers.

[0277] The specific connection medium between the memory 1601, the communication module 1603 and the processor 1602 is not limited in the embodiment of the present application. Fig.16 In the embodiment, the memory 1601 and the processor 1602 are connected via a bus 1604. The bus 1604 is connected to the processor 1602 via a bus 1604. Fig.16 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1604 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.16 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0278] The memory 1601 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the data lake management method of the embodiment of the present application. The processor 1602 is used to execute the above-mentioned data lake management method, such as Figure 2 or Figure 7 or Fig.10 shown.

[0279] In some possible implementations, various aspects of the data lake management method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the data lake management method according to various exemplary implementations of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 2 or Figure 7 or Fig.10 Follow the steps shown in .

[0280] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0281] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0282] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0283] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0284] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0285] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0286] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0287] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0288] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0289] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0290] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0291] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0292] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data lake management system, characterized in that: The system includes: a metadata management module, a big data fast processing module, and a query calculation engine management module; The metadata management module is used to provide a metadata management interface, and send a request for changing the library and table structure of the data lake triggered by an object through the metadata management interface to the query computing engine management module; The big data processing module is used to provide a big data processing interface, and send the data processing request of the data lake triggered by the object through the big data processing interface to the query calculation engine management module; The query computing engine management module is used to configure a specified query computing engine in the data lake and establish a resident resource pool through the query computing engine; after receiving the library table structure change request or the data processing request through the resident resource pool, start at least one target resident session in the resident resource pool through the query computing engine; and process the corresponding request through the at least one target resident session.

2. A data lake management method, applied to a query computing engine management module in a data lake management system, the method comprising: Configure a specified query computing engine in the data lake, and establish a resident resource pool through the query computing engine; After receiving a request for changing a library or table structure or a request for processing data for the data lake through the resident resource pool, starting at least one target resident session in the resident resource pool through the query computing engine; The library table structure change request is triggered by an object through a metadata management interface provided by a metadata management module in the data lake management system, and the data processing request is triggered by an object through a big data processing interface provided by a big data processing module in the data lake management system; The corresponding request is processed through the at least one target resident session.

3. The method according to claim 2, characterized in that The processing of the corresponding request by the at least one target resident session includes: Parsing the library table structure change request or data processing request through the at least one target resident session to obtain a structured query language SQL statement corresponding to the request processing; The structured query language is used to call the computing instance in the query computing engine to process the corresponding request.

4. The method according to claim 3, characterized in that The step of parsing the library table structure change request or data processing request of the data lake through the at least one target resident session further includes: For each target resident session, perform the following operations: Configure a preset survival check time for the target resident session, and periodically poll and run corresponding SQL statements according to the survival check time; If the target resident session can return normally, it indicates that the target resident session runs normally; If the target resident session fails to return normally, the target resident session is restarted through a restart mechanism.

5. The method according to claim 2, characterized in that The step of configuring a specified query computing engine in the data lake further includes: The resource allocation strategy of the query computing engine is configured so that the query computing engine dynamically allocates computing resources according to the request of the object.

6. The method according to claim 2, characterized in that The step of establishing a resident resource pool through the query computing engine includes: By configuring a preset number of resident sessions, a resident resource pool including multiple resident sessions is established to respond to library table structure change requests or data processing requests for the data lake in real time, and allocate target resident sessions according to the requests.

7. A data lake management method, applied to a big data processing module in a data lake management system, the method comprising: Provides a big data processing interface; A data processing request of the data lake triggered by the object through the big data processing interface is sent to a query computing engine management module in the data lake management system, so that after the query computing engine management module receives the data processing request through a resident resource pool, it starts at least one target resident session in the resident resource pool through a query computing engine, and processes it through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

8. The method according to claim 7, characterized in that The step of sending the data processing request of the data lake triggered by the object through the big data processing interface to the query computing engine management module in the data lake management system also includes: By configuring the big data processing cycle, the data lake management system can periodically schedule and process the big data in the data lake.

9. A data lake management method, applied to a metadata management module in a data lake management system, the method comprising: Provides metadata management interface; A request for changing the library and table structure of the data lake triggered by the object through the metadata management interface is sent to the query computing engine management module, so that after the query computing engine management module receives the request for changing the library and table structure through the resident resource pool, it starts at least one target resident session in the resident resource pool through the query computing engine, and processes it through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

10. A data lake management device, applied to a query computing engine management module in a data lake management system, characterized in that: include: A configuration unit, used to configure a specified query computing engine in the data lake, and establish a resident resource pool through the query computing engine; a starting unit, configured to start at least one target resident session in the resident resource pool through the query computing engine after receiving a library table structure change request or a data processing request for the data lake through the resident resource pool; The library table structure change request is triggered by an object through a metadata management interface provided by a metadata management module in the data lake management system, and the data processing request is triggered by an object through a big data processing interface provided by a big data processing module in the data lake management system; A processing unit is used to process the corresponding request through the at least one target resident session.

11. A data lake management device, applied to a big data processing module in a data lake management system, characterized in that: include: A first interface providing unit, used to provide a big data processing interface; The first sending unit is used to send a data processing request of the data lake triggered by the object through the big data processing interface to a query computing engine management module in the data lake management system, so that after the query computing engine management module receives the data processing request through a resident resource pool, it starts at least one target resident session in the resident resource pool through a query computing engine, and processes through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

12. A data lake management device, applied to a metadata management module in a data lake management system, characterized in that: include: The second interface providing unit is used to provide a metadata management interface; The second sending unit is used to send the library table structure change request of the data lake triggered by the object through the metadata management interface to the query computing engine management module, so that after the query computing engine management module receives the library table structure change request through the resident resource pool, it starts at least one target resident session in the resident resource pool through the query computing engine, and processes it through the at least one target resident session. The resident resource pool is established by the specified query computing engine configured in the data lake.

13. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 9.

Images