Distributed data lake system and performance optimization method

Abstract

The application discloses a distributed data lake system and a performance optimization method, and belongs to the technical field of cloud computing. The distributed data lake system provided by the application comprises a global metadata management component, which is used for hierarchically and uniformly managing metadata of a primary data lake node and at least one secondary data lake node; the primary data lake node stores a global metadata view, and uniformly manages the creation, modification and deletion of public metadata; the secondary data lake node is used for managing local private metadata; a distributed collaborative management network is composed of a collaborative agent module; the collaborative agent module is used for electing a master node in a local node cluster; the state information of the local node is registered to the primary data lake node, and the state information of other secondary data lake nodes is obtained from the primary data lake node; and a data mapping engine is used for mapping and processing original data from an edge database into fused data conforming to global specifications according to a predefined data model.

Description

Technical Field

[0001] This application belongs to the field of cloud computing technology, specifically relating to a distributed data lake system and its performance optimization method. Background Technology

[0002] As enterprises deepen their digital transformation, data lakes have become a critical infrastructure for integrating massive amounts of heterogeneous data from multiple sources. Traditional centralized data lake architectures typically employ a "storage-compute separation" model, consolidating raw data into a central storage location. In this model, any data analysis task spanning business domains or geographical regions requires transferring all relevant data from remote storage nodes to compute nodes via the network, resulting in significant network transmission overhead and data latency, severely limiting query response speed and real-time analysis capabilities.

[0003] Currently, to achieve cross-regional or cross-departmental data collaboration, related technologies typically employ distributed transaction mechanisms based on protocols such as Two-Phase Commit (2PC) for data synchronization and scheduling. Especially when a transaction spans multiple nodes in a data lake system, each node cannot know whether the operations of other nodes have succeeded or failed. In this case, a coordinator is usually selected from among the multiple nodes to uniformly control the operation results of all other nodes (participants) and ultimately instruct these nodes whether to actually commit the operation results. Specifically, in the preparation phase, participants notify the coordinator of the success or failure of their operations; in the execution phase, the coordinator decides whether each participant should commit or abort the operation based on feedback from all participants.

[0004] However, related technologies suffer from performance degradation due to low efficiency and high resource consumption in cross-domain data synchronization. This is because distributed transaction mechanisms based on the two-phase commit protocol have inherent flaws: firstly, a single point of failure in the coordinator can cause the entire data lake system to become blocked, resulting in poor reliability; secondly, coordinator failure can easily lead to inconsistent data states, requiring the introduction of additional consistency detection components, increasing the complexity and operational costs of the data lake system. These flaws cause cross-domain data lake systems to experience performance degradation when handling large-scale, high-concurrency queries due to low efficiency in cross-domain data synchronization and high resource consumption. Summary of the Invention

[0005] This application provides a distributed data lake system and a performance optimization method, which can solve the problem of reduced query performance caused by low efficiency and high resource consumption of cross-domain data synchronization in related technologies.

[0006] In a first aspect, embodiments of this application provide a distributed data lake system, including a primary data lake node and at least one secondary data lake node, the system further comprising: The global metadata management component, deployed on the primary data lake node, is used to perform hierarchical and unified management of metadata for the primary data lake node and at least one secondary data lake node. The primary data lake node stores the global metadata view and manages the creation, modification, and deletion of public metadata. The secondary data lake node is used to manage local private metadata. The distributed collaborative management network consists of collaborative agent modules deployed in each secondary data lake node. The collaborative agent module is used to elect a master node within its own node cluster, register its own status information with the primary data lake node, and obtain the status information of other secondary data lake nodes from the primary data lake node. The data mapping engine, deployed on the secondary data lake node, is used to map and process raw data from the edge database into fused data that conforms to global specifications, based on a predefined data model.

[0007] Secondly, embodiments of this application provide a performance optimization method for a distributed data lake system as described in the first aspect, comprising: After starting up and electing a master node, the secondary data lake node registers with the primary data lake node; the primary data lake node verifies the registration and triggers data synchronization, and synchronizes and caches cluster status information among all secondary data lake nodes; Based on a predefined data model, the raw data stored in the edge database is mapped and standardized and cleaned to obtain fused data that conforms to global specifications, and the fused data is persistently stored. For a query request, a coverage set is determined based on the statistical information of the query predicate. The final intersection is obtained by executing the subquery plan of the coverage set and taking the intersection of the results. The query request is then executed on the data file in the final intersection.

[0008] In some embodiments of this application, the distributed data lake system includes a primary data lake node and at least one secondary data lake node. The system further includes: a global metadata management component deployed on the primary data lake node for hierarchical and unified management of metadata for the primary data lake node and at least one secondary data lake node; wherein the primary data lake node stores a global metadata view and uniformly manages the creation, modification, and deletion of public metadata; the secondary data lake node manages local private metadata; a distributed collaborative management network composed of collaborative agent modules deployed within each secondary data lake node; wherein the collaborative agent module is used to elect a master node within its node cluster; register its own node's status information with the primary data lake node and obtain the status information of other secondary data lake nodes from the primary data lake node; and a data mapping engine deployed on the secondary data lake node for mapping and processing raw data from the edge database into fused data conforming to global specifications based on a predefined data model. Thus, in scenarios involving cross-domain query transactions, the distributed data lake system provided in this application embodiment utilizes a triple mechanism—reducing metadata overhead through a global metadata view, reducing redundant data migration through a distributed collaborative management network, and reducing data volume through raw data fusion—to minimize the resource consumption of cross-domain data synchronization. This significantly improves query response speed and overall system performance, solving the problem of reduced query performance caused by low efficiency and high resource consumption in related technologies. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic diagram of a distributed data lake system provided in some embodiments of this application; Figure 2A A schematic flowchart illustrating a performance optimization method for a distributed data lake system provided in some embodiments of this application; Figure 2B A schematic flowchart illustrating a performance optimization method for a distributed data lake system provided in some embodiments of this application; Figure 3A A schematic diagram of a data lake node in a distributed data lake system provided in some embodiments of this application; Figure 3B This is a schematic diagram illustrating inter-layer communication between any two data lake nodes in a distributed data lake system provided in some embodiments of this application. Figure 3C A schematic diagram of a distributed data lake system provided in some embodiments of this application; Figure 4A A schematic diagram illustrating the online / offline process of secondary and lower-level data lake nodes and the process of adding, deleting, and modifying clusters, provided for some embodiments of this application; Figure 4B The following are schematic diagrams illustrating the online / offline processes of secondary and lower-level data lake nodes and the processes for adding, deleting, and modifying clusters, provided for some embodiments of this application. Figure 4C This is a flowchart illustrating a cross-domain scheduling task provided in some embodiments of this application.

[0011] Explanation of reference numerals in the attached figures: 10 - Distributed Data Lake System; 100 - Primary Data Lake Node; 110 - Global Metadata Management Component; 120 - Distributed Coordination Service Component; 200 - Secondary Data Lake Node; 210 - Distributed Collaborative Management Network; 201 - Collaborative Agent Module; 220 - Data Mapping Engine; 230 - Query Optimization Execution Module; 300 - Data Lake Node; 310 - Business Application Layer; 320 - Data Analysis Layer; 330 - Computing Engine Layer; 340 - Resource Management Layer; 350 - Data Storage Layer; 360 - File Storage Layer. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0015] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application is explained below.

[0016] Data Lake: A data lake is a centralized repository used to store massive, multi-source, heterogeneous enterprise data (including structured, semi-structured, and unstructured data) in raw formats (such as raw files and binary large objects). Its core principle is "store first, define later," meaning that a strict schema does not need to be predefined when loading data, allowing users to process, analyze, explore, and mine the data on demand. Data lakes are typically tightly integrated with big data processing frameworks (such as Hadoop) and various analytical tools. For example, an enterprise-grade data lake built on Hadoop HDFS: A large company stores all its raw data, such as user clickstream logs, order transaction records, customer service voice text, and social media images, on HDFS, forming a massive data lake for subsequent user behavior analysis, recommendation algorithm training, etc. Another example is an integrated data lake warehouse product, which provides the flexibility of a data lake while introducing the data management and transaction capabilities of a data warehouse, allowing users to complete the entire process from raw data ingestion to high-performance analysis on a single platform.

[0017] Distributed Data Lake Systems: Distributed data lakes are an extension of the data lake concept in terms of physical deployment and architecture. Instead of storing all data centrally in a single physical location or cluster, they distribute and manage data across multiple storage clusters in multiple geographical locations, data centers, or cloud regions. These distributed storage nodes are logically integrated through unified metadata, namespaces, and a management plane, providing a unified view of data access. Driving forces include data locality compliance, reduced cross-domain transmission costs, and improved regional access performance. For example, an edge-cloud collaborative data lake: A nationwide smart city project deploys regional data lake nodes in various cities (edge ​​areas) for real-time storage and processing of local traffic camera video streams and sensor data. Simultaneously, a central data lake node is deployed on a provincial or national central cloud. After real-time filtering and aggregation of data by the regional data lake nodes, summaries or anomaly event data are synchronized to the central data lake node for macro-level analysis and decision-making.

[0018] For example, the cross-domain, hierarchical distributed data lake system of this application includes multiple data lake nodes. The architecture of a single data lake node includes a business application layer, a data analysis layer, a computing engine layer, a resource management layer, a data storage layer, and a file storage layer. That is, each data lake node is both a computing node and a storage node, making a single data lake node an integrated data lake warehouse product that takes into account both storage and computing functions. Users can complete the entire process from raw data ingestion to high-performance computing analysis on a single data lake node.

[0019] In a distributed data lake system, "cross-domain" (cross-node) refers to the fact that each data lake node (as a storage database) is located in a different storage domain. If a transaction spans multiple data lake nodes in the data lake system, the transaction can be called cross-domain or cross-node.

[0020] In a distributed data lake system, "hierarchical" refers to the following: multiple data lake nodes include data lake nodes at different levels. Data lake nodes at the same level can form a node cluster, and data lake nodes at different levels have a hierarchical relationship. For example, multiple data lake nodes may include first-level data lake nodes and multiple second-level data lake nodes. Multiple second-level data lake nodes can form a second-level data lake node cluster, and each second-level data lake node is subordinate to a first-level node. Of course, the number of levels in a distributed data lake system can be two or three. For example, multiple data lake nodes may also include multiple third-level data lake nodes, and multiple third-level data lake nodes can form a third-level data lake node cluster, and each third-level data lake node is subordinate to a second-level node. This application does not limit the number of levels in the distributed data lake system.

[0021] In this context, a primary data lake node refers to the logical center, global control, and collaboration core of the entire system in the cross-domain, hierarchical distributed data lake system architecture of this application.

[0022] In this context, a secondary data lake node refers to the actual data storage and computation entity subordinate to a primary node within the cross-domain, hierarchical distributed data lake system architecture described in this application. This secondary data lake node is responsible for data storage and computation within a specific region, business line, or data domain. It is a relatively complete data lake instance with independent storage and computing resources, physically storing raw data or refined, merged data, and handling local (edge ​​database) data governance and processing tasks. While receiving global collaboration from the primary data lake node, it possesses a high degree of autonomy.

[0023] Edge databases, in particular, are databases deployed on computing or storage nodes located at or near the source of data generation. They reside at the edge of the network topology, such as in retail stores, in-vehicle devices, IoT gateways, or mobile devices. Edge databases are used for localized, real-time or near-real-time processing, caching, filtering, and initial aggregation of data immediately after its generation. This reduces network transmission pressure on the central cloud, meets low-latency response requirements, and maintains local functionality during network outages.

[0024] To reduce the resource consumption of cross-domain data synchronization and improve the query efficiency of distributed data lakes, this application embodiment constructs a novel distributed data lake system that includes a global metadata management component, a distributed collaborative management network, and a data mapping engine. This system can systematically solve the problem of low efficiency in cross-domain data synchronization from an architectural perspective, thereby effectively improving query performance.

[0025] First, the global metadata management component provided in this application embodiment implements hierarchical and unified management of metadata. First-level data lake nodes store the global view and manage public metadata, while second-level data lake nodes independently manage private metadata. This design logically forms a unified virtual data view, allowing applications to transparently locate data based on global metadata. This avoids blind, cross-domain full metadata searches or synchronizations for data location, reducing unnecessary metadata network overhead.

[0026] Secondly, the distributed collaborative management network provided in this application elects a master node within the secondary data lake nodes through a collaborative agent module, and dynamically registers and synchronizes cluster status information with the primary data lake nodes. This mechanism establishes a decentralized, state-aware collaborative layer. When performing cross-domain queries, the system can intelligently decide the optimal execution path based on real-time and accurate node status (such as load, network distance, and data locality): whether to "sink" the computation to the node where the data resides, or to only synchronize the necessary "fused data." This achieves a shift from moving raw data to moving computation results or refined data on demand, fundamentally and significantly reducing the amount of data transmitted across domain networks, and lowering bandwidth consumption and transmission latency.

[0027] Finally, the data mapping engine provided in this application embodiment processes the original data in-situ into fused data that conforms to global specifications based on a predefined data model. This allows the data to undergo initial reduction and refinement near the source, and the data synchronized upstream or for cross-domain queries is already a lightweight fusion result, rather than a massive original dataset, further compressing the scale of data that needs to be synchronized.

[0028] In summary, the cross-domain, hierarchical distributed data lake system provided in this application utilizes a three-pronged mechanism—a logically unified global view to reduce metadata overhead, intelligent collaborative decision-making to reduce redundant data migration, and source data fusion to reduce data volume—to minimize cross-domain data synchronization resource consumption. This significantly improves query response speed and overall system performance, solving the problem of reduced query performance caused by low efficiency and high resource consumption in related technologies.

[0029] The distributed data lake system and performance optimization method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0030] Figure 1 A flowchart of a distributed data lake system provided for some embodiments of this application.

[0031] like Figure 1 As shown in the embodiment of this application, a distributed data lake system 10 may include: a primary data lake node 100 and at least one secondary data lake node 200.

[0032] The number of secondary data lake nodes (200) can be set according to actual needs, and this application does not impose any restrictions on this.

[0033] like Figure 1 As shown, the distributed data lake system 10 provided in this application embodiment also includes a global metadata management component 110; The global metadata management component 110 is deployed on the primary data lake node 100 and is used to perform hierarchical and unified management of the metadata of the primary data lake node 100 and at least one secondary data lake node 200. The primary data lake node 100 stores the global metadata view and manages the creation, modification, and deletion of public metadata in a unified manner; the secondary data lake node 200 is used to manage local private metadata. Public metadata refers to metadata shared by multiple data lake nodes, while private metadata refers to metadata used exclusively by a local (one) data lake node.

[0034] The global metadata management component 110 is used to maintain a globally unified metadata view and data routing information, enabling applications to transparently access data located in any data lake node.

[0035] In this embodiment, the global metadata management component 110 implements hierarchical and unified management of metadata. The first-level data lake node 100 stores the global metadata view and manages public metadata, while the second-level data lake nodes independently manage private metadata. This design logically forms a unified virtual data view, allowing applications to transparently locate data based on global metadata. This avoids blind, cross-domain full metadata searches or synchronizations for data location, reducing unnecessary metadata network overhead.

[0036] like Figure 1 As shown in the figure, the distributed data lake system 10 provided in this application embodiment also includes a distributed collaborative management network 210. The distributed collaborative management network 210 consists of collaborative agent modules 201 deployed within each secondary data lake node 200. The collaborative agent module 201 is used to: elect a master node within its own node cluster; register its own node's status information with the primary data lake node; and obtain the status information of other secondary data lake nodes from the primary data lake node. Here, "own node cluster" refers to a node cluster composed of at least one secondary data lake node 200 located at the same level.

[0037] In this embodiment, the distributed collaborative management network 210 elects a master node within at least one secondary data lake node 200 through the collaborative agent module 201, and dynamically registers and synchronizes cluster status information with the primary data lake node 100. This mechanism establishes a decentralized, state-aware collaborative layer. When performing cross-domain queries, the system can intelligently decide the optimal execution path based on real-time and accurate node status (such as load, network distance, and data locality): whether to "sink" the computation to the node where the data resides, or to only synchronize the necessary "fused state data". Compared with the handling of raw data in related technologies, this embodiment handles computation results or refined data on demand, fundamentally and significantly reducing the amount of data transmitted across domain networks, and lowering bandwidth consumption and transmission latency.

[0038] like Figure 1 As shown in the embodiment of this application, the distributed data lake system 10 also includes a data mapping engine 220. The data mapping engine 220 can be deployed on the secondary data lake node 200 and is used to map and process raw data from the edge database into fused data conforming to global specifications based on a predefined data model.

[0039] In this embodiment, the data mapping engine 220 deployed on the secondary data lake node 200 processes the raw data locally into fused data conforming to global specifications based on a predefined model. This allows the data to undergo initial reduction and refinement near the source, and the data synchronized upstream or for cross-domain queries is already a lightweight fusion result, rather than a massive original dataset, further compressing the scale of data that needs to be synchronized.

[0040] The distributed data lake system provided according to the embodiments of this application includes a primary data lake node and at least one secondary data lake node. The system further includes: a global metadata management component deployed on the primary data lake node, used for hierarchical and unified management of the metadata of the primary data lake node and at least one secondary data lake node; wherein, the primary data lake node stores a global metadata view and uniformly manages the creation, modification, and deletion of public metadata; the secondary data lake node is used to manage local private metadata; a distributed collaborative management network, composed of collaborative agent modules deployed in each secondary data lake node; wherein, the collaborative agent module is used to elect a master node within its node cluster; register its own node's status information with the primary data lake node and obtain the status information of other secondary data lake nodes from the primary data lake node; and a data mapping engine deployed on the secondary data lake node, used to map and process raw data from the edge database into fused data conforming to global specifications based on a predefined data model. Thus, in scenarios involving cross-domain query transactions, the distributed data lake system provided in this application embodiment achieves the minimization of cross-domain data synchronization resource consumption through a three-pronged mechanism: reducing metadata overhead through a global metadata view, reducing redundant data migration through a distributed collaborative management network, and reducing data volume through raw data fusion. This significantly improves query response speed and overall system performance.

[0041] In some embodiments of this application, in order to optimize the performance and cost of cross-domain data access through architecture-level collaboration and data refinement, in the distributed data lake system 10 provided in this application embodiment, the first-level data lake node 100 is used to store the governed fused data and global metadata index; the second-level data lake node 200 is used to store the original data from the edge database and the fused data obtained based on the mapping of the original data.

[0042] For example, primary data lake node 100 mainly stores master data, fused data that has been governed, and global metadata indexes; secondary data lake node 200 mainly stores raw data within the region, or fused data (micro-statistical data) obtained by mapping raw data; primary data lake node 100 accesses data in secondary data lake node 200 as needed through data routing.

[0043] The distributed data lake system 10 is used to perform cross-node data transfer or computation tasks based on the logical location of the data determined by the global metadata management component 110 and the real-time node status provided by the distributed collaborative management network 210.

[0044] In the distributed collaborative management network 210, each collaborative agent module 201 can also be used to: collaboratively execute cross-node data synchronization tasks and computing task scheduling based on the network-wide status information.

[0045] The distributed data lake system 10 provided in this application embodiment can clearly distinguish data storage roles and integrate an intelligent collaborative network, fundamentally solving the problems of large data transfer volume and high network consumption in traditional cross-domain data access.

[0046] For example, in one embodiment of this application, the data storage architecture of the distributed data lake system 10 may include a three-tiered data layer with separated roles: the first layer is the edge database, which is the data source and retains the original data. The second layer is the secondary data lake nodes, which are regional hubs that aggregate, map, standardize, and clean the local original data according to a predefined data model, and store micro-statistical data (fused data obtained based on the original data). The third layer is the primary data lake nodes, which do not store large amounts of original data, but instead store global metadata indexes, master data, and core fused data. The system centrally stores high-frequency, high-value refined data (such as master data and core fused data) and global metadata indexes in the primary data lake nodes, while storing large volumes of low-frequency micro-statistical data (such as fused data generated on demand) in the secondary data lake nodes. This hierarchical storage strategy based on data value and access patterns allows most analytical queries to be completed quickly in the data pool of the primary data lake nodes, avoiding unnecessary remote data access from a data layout perspective.

[0047] It should be noted that when cross-domain query transactions involve raw data from secondary data lake nodes, the system in this embodiment does not simply pull the full amount of data. Instead, the global metadata management component 210 first provides the logical location of the raw data, and then the distributed collaborative management network 210 provides the real-time node status of the system (such as the load, health status, network latency, etc. of each secondary data lake node). Based on these two types of real-time information, the system can make dynamic and optimal decisions: for computationally intensive tasks, computation is decentralized, scheduling the computation task to the secondary data lake node where the raw data is located, and only a simplified result is returned; for queries that require the fusion of multi-source data, the system intelligently selects the node combination with the shortest synchronization path and the lightest load, performing on-demand and precise data transfer. This achieves a shift from blind transmission to intelligent scheduling, greatly reducing redundant data transmission.

[0048] Furthermore, in this embodiment, the preprocessing of the original data at the edge further reduces the data volume of a single synchronization. The data mapping engine 220 deployed on the secondary data lake node 200 converts the original data into fused data that conforms to standard specifications according to the predefined data model when the original data is entered into the database. This ensures that even in scenarios requiring synchronization, the data transferred is highly compressed and aggregated fused data, rather than the original data, fundamentally reducing the data volume of a single synchronization.

[0049] In this way, the embodiments of this application, through the close integration of storage hierarchy, intelligent decision-making, and data refinement, systematically and multi-dimensionally reduce the network transmission overhead and computing latency required for cross-domain data access, thereby significantly improving query performance and greatly reducing costs.

[0050] In some embodiments of this application, in order to realize a decentralized and highly reliable cross-domain data lake node state synchronization and coordination mechanism, the distributed collaborative management network 210 can solve the problems inherent in traditional centralized coordination architectures (such as two-phase commit protocols) such as single point of failure, state inconsistency and recovery difficulties through node registration, state synchronization and information broadcasting mechanisms, thus providing underlying guarantees for the efficient and stable operation of cross-domain data lakes.

[0051] For example, in the distributed data lake system 10 provided in this application embodiment, the distributed collaborative management network 210 is specifically used to: manage the cluster status information of the secondary data lake nodes 200; wherein, after the secondary data lake node 210 is started, a master node is determined through an election mechanism, and the master node sends the registration information of the designated access node list to the primary data lake node 100; after verifying the registration information, the primary data lake node 100 triggers the synchronization of data and metadata with the secondary data lake node 200, and caches the cluster status information of the secondary data lake node 200; After the access node list information of the secondary data lake node 200 is successfully registered, the primary data lake node 100 caches the access node list information of the secondary data lake node 200 and synchronizes it to other registered secondary data lake nodes 200.

[0052] Among them, the list of designated access nodes for the Level 1 data lake node 100 is backed up in its own deployed distributed coordination service component.

[0053] In this embodiment, after each secondary data lake node starts up, it autonomously elects a master node through internal election and actively registers with the primary data lake node, eliminating absolute dependence on a fixed, centralized coordinator. The primary data lake node mainly undertakes the verification and forwarding functions of the registration center and information hub, rather than being a single coordinator for all transactions, thus architecturally mitigating the risk of network-wide congestion caused by the failure of the central coordinator. Simultaneously, the election mechanism ensures automatic succession of leadership within the secondary data lake nodes, further improving the availability of a single node cluster.

[0054] In this embodiment, after verifying the registration information, the primary data lake node not only caches the state itself but also synchronizes it to all other registered secondary data lake nodes. Combined with the primary data lake node's own access list information being backed up in a distributed coordination service component (such as ZooKeeper), an eventually consistent and highly redundant global state information database is constructed. This allows any node in the system to obtain a near-real-time, consistent global node topology and health status view based on its local or near-end cache when performing cross-domain task scheduling. This completely solves the problems of scheduling errors, duplicate data transfer, or task failures caused by inconsistent local views, providing a reliable data foundation for intelligent decision-making.

[0055] In this embodiment, when any node fails, its state change can be quickly broadcast to the entire network via the distributed collaborative management network 210. Other nodes can immediately exclude it from the scheduling target and trigger high-availability takeover (e.g., using a backup access node). After the failed node recovers, it can re-initiate the registration process and automatically integrate into the system, achieving rapid isolation and recovery from the fault. Node online / offline status changes or state changes (such as access list updates) are automatically synchronized through this standardized process, eliminating the need for manual cross-domain configuration. This frees operations and maintenance from complex manual intervention, greatly improving system maintainability and fault recovery efficiency.

[0056] Thus, by establishing a decentralized distributed collaborative management network 210 with self-registration of nodes, centralized verification of status, and network-wide broadcast synchronization, this embodiment of the application realizes automatic discovery, consistent view, and efficient synchronization of cross-domain data lake node status, providing stable and reliable underlying support for core functions such as data transfer and computing task scheduling at the upper layer, thereby improving the performance and reliability of the entire distributed data lake system.

[0057] In some embodiments of this application, in order to achieve real-time perception of the status of cross-domain data lake nodes and rapid self-healing of faults, this application embodiment introduces active heartbeat monitoring and a real-time status broadcasting mechanism based on RPC, which effectively solves the scheduling errors, resource waste and business interruption problems caused by the delay in status perception and slow fault propagation in traditional distributed systems.

[0058] For example, in the distributed data lake system 10 provided in this application embodiment, the distributed collaborative management network 210 is also used to: maintain heartbeat communication with the access node list of each secondary data lake node through the master control node to monitor the node status; when a node failure or recovery is detected, notify other secondary data lake nodes of the status change information through RPC (Remote Procedure Call) and update the cluster status information cached by other secondary data lake nodes.

[0059] In this embodiment, the continuous heartbeat communication maintained by the master node and the access node list constitutes the real-time monitoring layer of the system for node availability. This transforms the traditional fault detection mechanism that relies on timeout judgment or passive reporting into an active, periodic health check, thereby improving the detection time of node faults (such as process crashes or network partitions).

[0060] In this embodiment, once a node state change (failure or recovery) is detected, the system does not rely on slow broadcasting or polling, but immediately and reliably pushes the accurate change information directly to all other nodes that need to know this state via RPC. This ensures that the state view in the entire distributed collaborative management network 210 can reach final consistency in a very short time, avoiding errors such as "split-brain" problems or continuous task scheduling to failed nodes caused by inconsistent views of different nodes (e.g., some nodes believe a node is alive, while others believe it has failed).

[0061] In this embodiment, once a node failure is quickly detected and synchronized across the entire network, the scheduler can immediately remove it from the available resource pool and route subsequent tasks to healthy nodes. This significantly shortens the duration of service interruption due to a single point of failure, ensuring service continuity. Simultaneously, this avoids resource idleness and timeouts caused by continuously sending computing tasks or data synchronization requests to failed nodes, improving overall resource utilization. For recovered nodes, their status can also be quickly synchronized, allowing them to rapidly re-enter the resource pool and participate in load balancing, thus achieving dynamic and smooth scaling of system capacity.

[0062] Thus, by combining continuous heartbeat communication with RPC notifications, this embodiment of the application provides accurate and real-time decision-making basis for core functions such as data synchronization and task scheduling at the upper layer, enabling the entire distributed data lake system to quickly respond to topology changes, automatically avoid faults, and efficiently utilize resources, fundamentally enhancing the system's availability, resilience, and operational efficiency.

[0063] In some embodiments of this application, in order to provide dynamic and controllable distributed data lake node management and configuration capabilities, this application embodiment introduces a unified configuration interface to provide a centralized and programmable operation plane for the operation and maintenance management of the distributed data lake system, effectively solving the problems of low efficiency, error susceptibility and system rigidity caused by the reliance on manual and decentralized operation for node configuration changes in the traditional architecture.

[0064] For example, the distributed data lake system provided in this application embodiment also includes a configuration interface (not shown in the figure). The configuration interface is used to issue cluster management commands to the primary data lake nodes. The cluster management commands are used to add, delete or modify the registration information and access node list information of the secondary data lake nodes.

[0065] The configuration interface serves as a standardized interaction channel, allowing administrators to directly add, delete, or modify the registration information and access node list of secondary data lake nodes by issuing unified cluster management commands to the primary data lake nodes. This completely changes the cumbersome process of manual configuration on each node in the traditional model, converging the originally scattered and complex configuration management work into a single control point, greatly improving the convenience and efficiency of management operations.

[0066] For example, when business needs to scale horizontally to cope with increased load, administrators can quickly add new secondary data lake nodes (registration information and access list) through the configuration interface. The system will automatically trigger the aforementioned registration, status synchronization, and data synchronization processes, allowing the new node to seamlessly and quickly integrate into the existing cluster and begin providing services, achieving "hot-swappable" expansion of computing and storage resources. Conversely, when a node is taken offline, maintained, or replaced, its configuration can also be securely deleted or modified through the interface. The system will automatically update the global status view and adjust the task scheduling strategy to ensure a smooth service transition and avoid the risk of service interruption or data inconsistency caused by node changes.

[0067] Furthermore, the standardization of configuration interfaces makes it easy to integrate with external operations and maintenance platforms, orchestration systems, or infrastructure-as-code tools. Combined with automation scripts or policy engines, advanced automation scenarios such as dynamic scaling and node migration based on load thresholds, cost optimization, or compliance requirements can be achieved. This frees operations and maintenance personnel from repetitive manual operations, reduces the possibility of human error, and enables the system to respond more intelligently and agilely to business needs and infrastructure changes.

[0068] Thus, by providing a centralized configuration interface, this embodiment of the application improves operational efficiency and ensures the long-term maintainability and business agility of the large-scale distributed data lake system in complex and ever-changing environments.

[0069] In some embodiments of this application, in order to ensure that the distributed coordination function is automatically restored and the state is consistent when the master control node fails, the embodiments of this application deploy a distributed coordination service component on the core primary data lake node and use the distributed coordination service component to ensure the failure recovery of the master control node.

[0070] For example, such as Figure 1 As shown in the figure, the distributed data lake system 10 provided in this application embodiment also includes a distributed coordination service component 120. The distributed coordination service component 120 can be deployed on the primary data lake node 100. The distributed coordination service component 120 can be used to: re-elect a new primary control node in the event of a failure of the primary control node of the secondary data lake node 200; wherein the new primary control node is used to obtain the backed-up cluster state information from the distributed coordination service component 120 and re-cache it.

[0071] In order to restore the cross-domain collaboration function, the new master node obtains the backed-up cluster status information from the distributed coordination service component 120 and re-caches it.

[0072] In this embodiment, key status information is externally stored in the distributed coordination service component 120. When the active master node of the secondary data lake node 200 fails, its node cluster no longer relies on the local cache or memory status of the failed node. Instead, a new master node is quickly and automatically elected through the election algorithm built into the distributed coordination service component 120, strictly avoiding the risk of "split-brain" and ensuring the uniqueness of leadership.

[0073] The newly elected master node can read the previously backed-up, complete cluster state information (including global node topology, access list, etc.) from the trusted external data source, the distributed coordination service component 120, and reload it into its local cache, ensuring strong consistency and high reliability in fault recovery. The system state is no longer lost or enters an uncertain state due to the loss of individual nodes. The new master node can immediately obtain the system view before the fault and use it as a basis to reconstruct all the context information required for cross-domain collaboration.

[0074] In this embodiment, by introducing a distributed coordination service component 120 to re-elect a new master node, the new master node obtains the backed-up cluster state information from the distributed coordination service component and re-caches it. This ensures that even after a master node failure switch, the entire distributed data lake system's perception of the cluster state remains continuous and consistent. Upper-layer data synchronization and task scheduling logic can be seamlessly integrated, preventing errors caused by gaps or rollbacks in the state view. The entire process is automatically triggered by the system without manual intervention, thereby achieving high availability for cross-domain collaborative services and significantly improving the system's self-healing capabilities.

[0075] In some embodiments of this application, in order to achieve automated, standardized integration and governance of multi-source heterogeneous data at the edge of the data lake, this application introduces a data mapping engine to bring data governance and standardization work forward to the edge of the data lake. This edge-to-governance model effectively prevents the data swamp problem that may be caused by raw data directly flooding into the central lake, significantly improves the quality and availability of the data entering the database, and fundamentally changes the traditional data lake model of batch ETL processing at the central node.

[0076] For example, the data mapping engine 220 is used to: determine the pattern matching rules from the edge database to the secondary data lake node based on the predefined data entity model and data mapping model; map and aggregate the raw data from the edge database to the secondary data lake node based on the pattern matching rules; standardize and clean the raw data according to the standardization specifications to obtain fused data that conforms to the global specifications, and persist the fused data.

[0077] In this embodiment, the data mapping engine 220 performs pattern matching, transformation, and aggregation on raw data from various heterogeneous edge databases based on a unified, predefined data entity model and mapping model. This ensures that the data has undergone preliminary cleaning, format unification, and business meaning alignment before entering the secondary data lake node, generating fused data that can be directly used for analysis.

[0078] In this design, the data stored and synchronized at the secondary data lake nodes is no longer the massive amount of raw data, but rather refined semi-finished data (fused data / micro-statistics), significantly compressing the amount of data that needs to be transmitted across the network. This not only greatly reduces network bandwidth consumption and synchronization latency, but also enables the primary data lake nodes to obtain refined information that can be used for decision-making more quickly. Simultaneously, locally at the secondary data lake nodes, the standardized and cleaned fused data can directly support regional real-time or near-real-time analysis needs, achieving on-site data processing and value generation, and alleviating the computational pressure on the central primary data lake nodes.

[0079] In this way, the embodiments of this application use a data mapping engine to perform pattern matching, transformation and aggregation on the raw data from various heterogeneous edge databases, thereby improving data quality, significantly reducing network bandwidth consumption and synchronization latency, and enabling primary data lake nodes to obtain refined information that can be used for decision-making more quickly. This achieves the core technical support for the entire cross-domain data lake system to reduce transmission costs, accelerate analysis timeliness and support edge intelligence.

[0080] In one embodiment of this application, in order to achieve accurate and efficient conversion from heterogeneous data to fused data through matrix transformation and normalization, the pattern matching rules may include: data source definition, target definition, table mapping relationship, field mapping relationship, data update mechanism and condition combination.

[0081] The data mapping engine 220 is specifically used for: mapping and aggregating raw data to secondary data lake nodes based on pattern matching rules; processing the data in the secondary data lake nodes into a data matrix; standardizing the data matrix according to standard data specifications to generate a standardized matrix; and cleaning the data in the standardized matrix to form fused data that conforms to global specifications.

[0082] In this embodiment, the data mapping engine 220 abstracts the aggregated data into a data matrix. This mathematical abstraction unifies complex, nested data structures (such as tables, fields, and records) into a standardized form of rows, columns, and values, shielding the physical storage and structural differences of the underlying heterogeneous data sources. This allows subsequent batch operations such as standardization and cleaning to be executed efficiently based on a unified matrix operation logic, greatly simplifying the processing logic and providing a foundation for algorithm optimization (such as vectorized computation).

[0083] In this embodiment, the standardization matrix generation step, by applying predefined global standard data specifications (such as field names, types, and encoding rules), forcibly aligns column vectors representing different semantics in the matrix to a unified metadata definition, achieving semantic layer fusion of cross-source data. Data cleaning based on this foundation applies unified quality rules (such as deduplication, error correction, and missing value filling) to the standardized columns, thereby efficiently producing fused data that conforms to global specifications. This phased standardization and cleaning process ensures data quality and consistency, significantly reducing the complexity and error probability of data integration.

[0084] In this way, the data mapping engine 220 achieves efficient and accurate transformation from raw data to fused data by matrix-based abstraction and integration of heterogeneity, standardized processing of unified semantics, and pipeline operation to ensure quality. It also makes the entire data preparation process predictable, measurable and easy to maintain.

[0085] In some embodiments of this application, the distributed data lake system 10 further includes a query optimization execution module 230, which is used to receive query requests, determine a coverage set based on the statistical information of the query predicates, obtain the final intersection by executing the subquery plan of the coverage set and taking the intersection of the results; and execute query requests on the data files in the final intersection.

[0086] The query optimization execution module 230 can be deployed within the secondary data lake node 200.

[0087] Of course, in other embodiments, the query optimization execution module may also be deployed within a primary data lake node (not shown in the figure), and this application does not impose any restrictions on this.

[0088] It should be noted that, for query requests, the query optimization execution module 230 can transform the traditional full data scan into precise coverage set addressing. Upon receiving a query request, the query optimization execution module 230 does not directly access all relevant data files. Instead, it first parses the filtering conditions (i.e., predicates) in the query and, based on system-maintained statistical information (such as data distribution, indexes, and file granularity statistics), calculates the minimum set of data files required to access only to satisfy a single or partial predicate, i.e., the coverage set. Then, by executing subqueries of these coverage sets and taking the intersection of the results, redundant files that do not contain the target data are fundamentally filtered out. Ultimately, only the few necessary data files pointed to by the intersection need to be executed with the complete query request. This process focuses computational resources on truly relevant data fragments.

[0089] By avoiding the reading and analysis of large amounts of irrelevant data files, disk I / O, network transmission (in the case of cross-node queries), and CPU computational overhead are drastically reduced, significantly lowering resource consumption and improving query response time by orders of magnitude. This is especially suitable for interactive analysis scenarios involving massive amounts of data files.

[0090] Meanwhile, precise file access also means less temporary data generation and memory consumption, improving the overall throughput and stability of a single node in concurrent query scenarios. This optimization method is adaptive, and its effect becomes more significant as the amount of data increases and the predicate selectivity improves.

[0091] In this way, the query optimization execution module 230 shifts the query performance bottleneck from the data access level to the more efficient predicate logic analysis level through the optimized execution path of cover set calculation, subquery intersection, and precise execution, thereby achieving the optimal balance between query efficiency and resource utilization in a distributed data lake environment.

[0092] For example, in order to optimize the execution path of complex queries through the optimization plan selection of the cost model, the query optimization execution module 230 is specifically used for: Parse the filter conditions in the query request and decompose the filter conditions into multiple logical sub-conditions; For at least one of multiple logical sub-conditions, based on the statistical information of the data lake system, determine the set of data files that need to be accessed to satisfy at least one logical sub-condition, and obtain the coverage set; By combining different coverage sets to form candidate execution plans, the candidate execution plan with the lowest estimated cost is selected to execute the query request.

[0093] For example, first, the WHERE clause of the query request is converted into conjunctive normal form (CNF); then, for the predicate subset in CNF, based on data lake statistics, the number of records that satisfy the predicate subset and the corresponding file coverage set are determined; finally, with the goal of minimizing the total query execution cost, the optimal combination of predicate subsets and their coverage set are selected to execute the query request.

[0094] One method for estimating the number of records that satisfy the predicate subset is to treat the estimated number of records as random balls, treat the data files as boxes, and estimate the number of non-empty boxes as the number of covered files using a statistical model.

[0095] In this embodiment, by decomposing the filtering conditions into logical sub-conditions and determining their corresponding file coverage sets, the system can accurately quantify the physical I / O cost of executing different combinations of subqueries. The mapping between record counts and files is abstracted as a "ball and box" model, and statistical information is used to estimate the coverage set size, providing a comparable quantitative cost estimation basis for each candidate plan. This allows the system to abandon a single, fixed execution path and instead dynamically search for the optimal solution within a solution space composed of different coverage set combinations, based on estimated costs (such as the sum of I / O, network, and CPU overhead). Especially when facing complex and variable query conditions and massive data distributions, the system no longer relies on potentially ineffective empirical rules but instead performs adaptive optimization based on real-time statistical information, ensuring that each query selects a (near) optimal execution plan under known information, significantly improving the stability and optimality of query performance.

[0096] Thus, this application embodiment transforms query optimization into a dynamic programming problem based on quantified costs, thereby continuously delivering high-performance and predictable query services in complex distributed environments.

[0097] This application also provides a performance optimization method for a distributed data lake system based on any of the above system embodiments.

[0098] like Figure 2A As shown, the performance optimization method provided in this application embodiment may include: Step 204: After starting up and electing a master node, the secondary data lake node registers with the primary data lake node; the primary data lake node verifies the registration and triggers data synchronization, and synchronizes and caches cluster status information among all secondary data lake nodes; it monitors the health status of nodes through a heartbeat mechanism and performs global notification updates when the status changes; Step 206: Based on the predefined data model, the original data stored in the edge database is mapped and standardized and cleaned to obtain fused data that conforms to the global specifications, and the fused data is persistently stored.

[0099] Step 208: For the query request, determine the coverage set based on the statistical information of the query predicate, obtain the final intersection by executing the subquery plan of the coverage set and taking the intersection of the results; execute the query request on the data file in the final intersection.

[0100] Thus, in scenarios involving cross-domain query transactions, the performance optimization method provided in this application embodiment works synergistically through a triple mechanism: reducing metadata overhead through a global metadata view, reducing redundant data migration through a distributed collaborative management network, and reducing data volume through raw data fusion. This minimizes the resource consumption for cross-domain data synchronization, thereby significantly improving query response speed and overall system performance.

[0101] In a specific example, step 204 above is a cross-domain collaborative management step, which may include: After the secondary data lake node is started, the collaborative agent module deployed in the secondary data lake node determines the master node through an election mechanism within the node cluster; The master control node sends the registration information of the specified access node list to the primary data lake node; After verifying the registration information, the primary data lake node triggers the synchronization of data and metadata with the secondary data lake node, and caches the cluster status information of the secondary data lake node; After the access node list information of the secondary data lake node is successfully registered, the primary data lake node will cache the access node list information of the secondary data lake node and synchronize it to other registered secondary data lake nodes. The master node maintains heartbeat communication with the access node list of each secondary data lake node to monitor node status; When a node failure or recovery is detected, the state change information is notified to other secondary data lake nodes via remote procedure call, updating the cluster state information cached by the other secondary data lake nodes.

[0102] Thus, this embodiment of the application achieves automatic discovery, consistent view and efficient synchronization of cross-domain data lake node status through decentralized node self-registration, centralized status verification and network-wide broadcast synchronization. This provides stable and reliable underlying support for core functions such as data transfer and computing task scheduling at the upper layer, and improves the performance and reliability of the entire distributed data lake system.

[0103] Furthermore, by introducing active heartbeat monitoring and a real-time status broadcasting mechanism based on RPC, the embodiments of this application effectively solve the problems of scheduling errors, resource waste, and service interruptions caused by the delay in status awareness and slow fault propagation in traditional distributed systems.

[0104] In a specific example, such as Figure 2BAs shown, after step 204 above, the performance optimization method provided in this application embodiment may further include: Step 205: In the event of a failure of the master node in the secondary data lake node, the distributed coordination service component re-elects a new master node; the new master node retrieves the backed-up cluster state information from the distributed coordination service component and re-caches it.

[0105] Thus, this embodiment of the application introduces a distributed coordination service component to re-elect a new master node. The new master node obtains the backed-up cluster state information from the distributed coordination service component and re-caches it, ensuring that even after a master node failure switch, the entire distributed data lake system's perception of the cluster state remains continuous and consistent. Upper-layer data synchronization and task scheduling logic can be seamlessly integrated, preventing errors caused by gaps or rollbacks in the state view. The entire process is automatically triggered by the system without manual intervention, thereby achieving high availability for cross-domain collaborative services and significantly improving the system's self-healing capabilities.

[0106] In a specific example, step 206 above is a data processing step, which involves mapping and processing the raw data stored in the edge database into fused data that conforms to global specifications, and then persistently storing it after standardization and cleaning. Specifically, this may include: Construct a pattern matching rule matrix from the edge database to the secondary data lake nodes; The original data in the edge database is mapped using a pattern matching rule matrix to obtain the data matrix expression in the secondary data lake node; The data matrix expression is standardized according to the standard data specification to obtain the data standardization matrix; Data is cleaned and persistently stored based on a data standardization matrix.

[0107] In this way, by abstracting and integrating heterogeneity through matrix-based methods, standardizing semantic processing, and ensuring quality through pipeline operations, not only is the efficient and accurate transformation from raw data to fused data achieved, but the entire data preparation process also becomes predictable, measurable, and easy to maintain.

[0108] In a specific example, step 208 above is an optimized execution step for the query request, which may include: estimating and selecting the optimal file coverage set to execute the sub-plan based on predicate statistics; and executing the final query only on the smallest file set by taking the intersection of the sub-plan results.

[0109] It should be noted that the process of estimating and selecting the optimal file coverage set is formalized as an optimization problem, and then a heuristic algorithm is used to solve this optimization problem in order to minimize the estimated total query execution time.

[0110] In this way, by optimizing the path of covering set computation, subquery intersection, and precise execution, the query performance bottleneck is shifted from the data access level to the more efficient predicate logic analysis level, thereby achieving the optimal balance between query efficiency and resource utilization in a distributed data lake environment.

[0111] In a specific example, such as Figure 2B As shown, prior to step 204 above, the performance optimization method provided in this application embodiment may further include: Step 202: Establish a global metadata management component deployed on the primary data lake node to perform hierarchical and unified management of metadata for the primary data lake node and at least one secondary data lake node.

[0112] The primary data lake nodes store the global metadata view and manage the creation, modification, and deletion of public metadata in a unified manner; the secondary data lake nodes are used to manage local private metadata.

[0113] The global metadata management component is used to uniformly and hierarchically manage cross-domain metadata; the first-level data lake node is used to store the global metadata view across domains (i.e., across nodes) and uniformly control the creation, modification, and deletion of public metadata; the second-level data lake node (and data lake nodes below the second level) is used to manage the private metadata of the local center.

[0114] The data lake nodes at each level are physically independent and have efficient access to metadata services: the operation and storage of private metadata of each secondary data lake node are independent of each other; through two-layer metadata access, data can be located quickly and accurately, and the hierarchical storage model can provide efficient metadata services.

[0115] The metadata of data lake nodes at all levels of the system can be distributed and stored on HDFS, which continues the high reliability and data security of Hadoop storage and ensures the secure storage of metadata of data lake nodes at all levels of the system.

[0116] Thus, this embodiment of the application achieves hierarchical and unified management of metadata through a global metadata management component. First-level data lake nodes store the global metadata view and manage public metadata, while second-level data lake nodes independently manage private metadata. This design logically forms a unified virtual data view, allowing applications to transparently locate data based on global metadata. This avoids blind, cross-domain full metadata searches or synchronizations for data location, reducing unnecessary metadata network overhead.

[0117] In practical applications, the traditional cross-domain data lake (warehouse) deployment method has evolved from two-site three-center to distributed multi-center. However, the embodiments of this application break through the traditional two-site three-center or distributed multi-center construction methods and propose a hierarchical data lake system architecture that includes a primary data lake node and at least one secondary data lake node. See details. Figure 3A , Figure 3B and Figure 3C .

[0118] like Figure 3A As shown, in the distributed data lake system provided in this application embodiment, the data lake node 300 may include the following components for the structure of each data lake node: a business application layer 310, a data analysis layer 320, a computing engine layer 330, a resource management layer 340, a data storage layer 350, and a file storage layer 360.

[0119] In this embodiment, the data lake node 300 can be any data lake node in the distributed data lake system. The data lake node 300 can be a first-level data lake node, a second-level data lake node, a third-level data lake node, etc. The implementation of this application does not limit the level to which the data lake node 300 belongs.

[0120] Thus, in the distributed data lake system provided in this application embodiment, each data lake node is both a computing node and a storage node, making a single data lake node an integrated data lake warehouse product that takes into account both storage and computing functions. Users can complete the entire process from raw data ingestion to high-performance computing analysis on the same data lake node.

[0121] like Figure 3B As shown, in the distributed data lake system provided in this application embodiment, any two data lake nodes of the same type can communicate with each other.

[0122] To avoid redundancy, Figure 3B The internal structure of data lake node 2 shown can be the same as that of data lake node 1. For upper-layer applications (APPs), both local and remote data analytics layers can be accessed transparently. For example, ... Figure 3B As shown, during the communication process between the data analysis layer of data lake node 1 and the data analysis layer of data lake node 2 through the external communication protocol, the upper-layer application (APP) transparently sends SQL or query requests (regardless of whether the calculation is completed by the data analysis layer within this node or by the data analysis layers of multiple internal and external nodes through mutual communication), which can be interpreted and executed by the data analysis layer protocol component.

[0123] In addition, such as Figure 3B The communication between other layers of the same type shown can be analogous to providing transparent access to the corresponding local and remote layers above them.

[0124] according to Figure 3A The internal structure of the data lake node shown is as follows: Figure 3B Inter-layer communication between corresponding layers of the data lake nodes is shown, and a method is proposed as follows: Figure 3C The distributed data lake system shown is an example. Figure 3C As shown, a distributed data lake system includes a primary data lake node and at least one secondary data lake node. Figure 3C The example illustrates two secondary data lake nodes: secondary data lake node A and secondary data lake node B.

[0125] The distributed data lake system can be viewed as a global database. The storage domain of the distributed data lake system includes a primary data lake node, a secondary data lake node A, and a secondary data lake node B.

[0126] Among them, the primary data lake node can be regarded as an integrated lake warehouse platform, used for unified management of cross-domain metadata, hierarchical storage of cross-domain metadata, unified metadata version management, and unified metadata write lock management.

[0127] The secondary data lake node can include a secondary regional database for data storage services and metadata management.

[0128] In this context, whether it is a primary data lake node or a secondary data lake node, the database objects of each data lake node include, but are not limited to, tables, views, and stored procedures.

[0129] Among them, the first-level data lake node is used for cross-domain metadata management, so that the data at each layer has an independent management and scheduling mechanism; In this model, the operation and storage of private metadata of each secondary data lake node are independent of each other; through two-layer metadata access, data can be located quickly and accurately, and the hierarchical storage model can provide efficient metadata services. Among them, the metadata of each center of the cross-domain big data platform is stored securely: the metadata of each data lake node is distributed and stored on HDFS, which continues the high reliability and data security of Hadoop storage and realizes the secure storage of the metadata of each data lake node.

[0130] In this embodiment, metadata defines the storage domain information to which the data belongs, thereby locating the data's storage location; the first-level data lake node stores a global view of metadata and the node's private metadata; the second-level data lake node stores complete metadata content, such as the logical file location of the data, the actual storage location of the data blocks, and the data organization structure. In this embodiment, the advantage of establishing a two-tier metadata synchronization mechanism lies in ensuring the synchronization of public metadata across both tiers by leveraging global transaction control mechanisms such as MVCC and prioritizing the management rights of the primary data lake nodes. Metadata CRUD operations are performed within independent transactions, guaranteeing metadata consistency between the two data centers.

[0131] In this embodiment, data lake nodes at all levels have a unified metadata standard: the unified standard based on CWM enables the sharing and exchange of metadata; the unified specification based on CWM enables general and scalable metadata storage; and the unified premise based on CWM enables unified management of metadata.

[0132] In this embodiment, the improvement of the distributed data lake system lies in the addition of cross-domain service components to the existing big data platform (big data lake) based on the Hadoop ecosystem. These cross-domain service components include, but are not limited to, a task scheduling engine, task collaboration services, metadata, access control, and monitoring services.

[0133] In this embodiment, the distributed data lake system includes a primary level and its secondary and lower-level data lake nodes, with each level platform constructed as a distributed cluster architecture. To ensure the smooth operation of cross-domain functions such as task scheduling, task collaboration, data and metadata synchronization, and divide-and-conquer within and between each distributed cluster, a cross-domain cluster status information management system specifically designed for this scenario is introduced to ensure efficient and stable data processing and collaboration.

[0134] In this embodiment of the application, any node in any data lake is used as the entry node for data processing. When the scheduling engine executes the scheduling task, it needs to be able to obtain the cluster status information of the specific data lake. Moreover, this information must be consistent in real time on any node in the cluster to ensure the real-time consistency of the cluster for data queries.

[0135] In this embodiment of the application, during the initial setup phase of the distributed data lake system, the cluster status information of all current data lakes is obtained, thereby coordinating the cluster information of each data lake and performing data and metadata synchronization and divide-and-conquer.

[0136] To address the shortcomings of traditional big data platforms, the distributed data lake system provided in this application embodiment is constructed using the following steps: First, a distributed coordination service component is deployed within a cluster consisting of multiple data lake nodes at the same level for task collaboration within the cluster. Each node in the cluster registers as a client of the distributed coordination service component, listens for changes to multiple file directories of the distributed coordination service component, and processes callback messages accordingly.

[0137] When building a cross-domain scenario, each level of data lake needs to provide a clear list of publicly accessible nodes to support the effective implementation of cross-domain functions. Specifically, the publicly accessible node list for the first-level data lake is named "MASTER_ACCESS_LIST" (i.e., the master node access list), while the publicly accessible node lists for each second-level and lower-level data lake are named "SLAVE_ACCESS_LIST" (i.e., the slave node access list). Simultaneously, to ensure the reliability and security of node information, key status information such as the server address, open access ports, and connectability status of these nodes is backed up in their respective distributed coordination service components. By opening multiple nodes, the system can automatically switch to other available nodes when a single node fails, thus ensuring the continuous and stable operation of cross-domain functions.

[0138] After the secondary and lower-level data lakes are successfully launched, a strict election mechanism will be used to determine the cluster's master node (i.e., the leader node). This master node is used to send registration requests to the primary-level "MASTER_ACCESS_LIST" (i.e., the master node access list) at preset time intervals via Remote Procedure Call (RPC) to ensure its legitimacy and validity.

[0139] After the primary data lake successfully starts up and elects a cluster master node (i.e., leader node), the primary level's "MASTER_ACCESS_LIST" (master node access list) can receive registration information from secondary and lower-level nodes. This information will first be verified to confirm whether it belongs to a known data lake identity. Once the identity of the secondary and lower-level nodes is confirmed, the system will further verify the legitimacy of their secondary level's "SLAVE_ACCESS_LIST" (slave node access list). After verification, the primary level will send data and metadata synchronization messages to the secondary and lower levels and initiate the synchronization process.

[0140] Once the synchronization process is successfully completed, Level 1 will take a series of measures to ensure data integrity and security. Specifically, Level 1 will back up the information of Level 2 and below, along with their SLAVE_ACCESS_LIST, to the distributed coordination service component, and simultaneously cache one copy of the data on the leader node of this cluster, and only one copy. This step marks the formal completion of the registration process of the Level 2 and below platforms with Level 1.

[0141] Each successfully registered second-level and lower-level data lake node can send an RPC notification to inform it of the addition of a new platform to the system. Simultaneously, the latest cached information across all cross-domain platforms is updated synchronously through the SLAVE_ACCESS_LIST of each second-level and lower-level node to ensure that each second-level and lower-level platform can obtain the latest data status in a timely manner.

[0142] Each secondary and lower-level data lake node's corresponding "SLAVE_ACCESS_LIST" (i.e., the list of nodes that have joined from the primary node) has received the latest cross-domain platform cache information sent by the primary node and has backed it up in the distributed coordination service component. Meanwhile, to ensure data consistency and efficiency, only one copy of this information is cached on the leader node of this cluster, signifying that the secondary and lower-level nodes have successfully completed the registration process across the entire cross-domain cluster. Once registration is successful, any node on any platform can access the complete cross-domain platform cluster management information cached on the leader node of this platform in real time via RPC, including the SLAVE_ACCESS_LIST information of newly registered secondary and lower-level nodes. The real-time availability of this information will support the smooth execution of cross-platform tasks such as cross-domain scheduling.

[0143] Specifically, for successfully registered second-level and lower-level data lake nodes, they are defined as being online on the cross-domain platform (unregistered platforms are considered offline). The leader node of an online second-level and lower-level data lake node continuously maintains RPC heartbeat communication with the MASTER_ACCESS_LIST of the first-level data lake node, while also supporting visualization to ensure real-time monitoring of the normal operation of cross-domain functions. Similarly, the first-level leader node also maintains RPC heartbeat communication with the SLAVE_ACCESS_LIST of each second-level and lower-level platform, providing visualization to monitor the stable operation of cross-domain functions. This ensures that the status of "MASTER_ACCESS_LIST" (i.e., the master node access list) and "SLAVE_ACCESS_LIST" (i.e., the slave node access list) can be monitored in real time at both the second-level and lower-level and the first-level levels, achieving accurate and timely information sharing, thereby significantly improving the system's flexibility and responsiveness.

[0144] In addition, for unified management of secondary data lake nodes, the system also includes a configuration interface. Primary data lake nodes use this interface to issue cluster management commands to manage secondary data lake nodes. For example, cluster management commands include the registration command "CREATE CLUSTER," used to allow secondary and lower-level data lake nodes to register with the distributed data lake system; and the rejection command "DROP CLUSTER," used to reject the registration of a specific secondary or lower-level data lake node. Cluster management commands can also include the modification command "ALTER CLUSTER," used to change the "SLAVE_ACCESS_LIST" (i.e., the slave node access list) of a specified secondary or lower-level data lake node. These actions can be executed on any node in the primary cluster. This cluster management information is backed up in the distributed coordination service component and cached, with a copy also cached on the master node of this cluster.

[0145] For example, refer to Figure 4A After the primary node receives the cluster management information, the online / offline processes and the processes for adding, deleting, and modifying cluster data lake nodes at the secondary and lower levels include: at the primary node (e.g., Figure 4A Upon receiving changes to the cluster management information configured by the user through the configuration interface, Node 1, Node 2, or Node 3 immediately initiates an RPC notification to each successfully registered secondary and lower-level data lake node. Through the "SLAVE_ACCESS_LIST" (i.e., the slave node access list) of each secondary and lower-level data lake node, it synchronously updates the latest cached information of each data lake node. Upon receiving changes to the cluster management information, the "SLAVE_ACCESS_LIST" (i.e., the slave node access list) of the secondary and lower-level data lake nodes backs it up in the distributed coordination service component of the primary node and caches only one copy in the master node of their respective cluster.

[0146] For example, refer to Figure 4B After receiving cluster management information, the online / offline processes and cluster addition / deletion / modification processes for data lake nodes at the second level and below may include: at the second-level data lake node (e.g., ... Figure 4B After receiving cluster management information, node 2) is backed up in the distributed coordination service component of the primary node and cached only once in the master node of this cluster.

[0147] For example, refer to Figure 4C Data lake nodes at the same level can also share extended information such as cluster load information for more complex cross-domain scheduling tasks. Specifically, ordinary nodes within the cluster (i.e., non-master nodes within the cluster, such as...) Figure 4C Nodes 2 and 3 shown (as shown) communicate via RPC to the master node of this cluster (e.g., ...). Figure 4C Node 1 shown can obtain cached cluster management information in real time.

[0148] In addition, to ensure the stability of the system, this application embodiment also provides fault and recovery handling mechanisms for each component.

[0149] For example, if a node in the "MASTER_ACCESS_LIST" (i.e., the master node access list) fails, the other surviving nodes in the "MASTER_ACCESS_LIST" can still continue to perform cross-domain tasks. As mentioned above, the "MASTER_ACCESS_LIST" information has been backed up in the distributed coordination service component. Since the master node in the cluster is a client of the distributed coordination service component, upon receiving this message notification, it changes the status of the failed node to an unreachable state and broadcasts this change via RPC to the "SLAVE_ACCESS_LIST" (i.e., the slave node access list) of the second-level and lower-level node clusters. This information is also backed up in the distributed coordination service component and cached on the master node of the second-level and lower-level node clusters. In this way, all nodes in the entire cross-domain cluster can visually display the failed node. After the failed node recovers, the above process can be automatically triggered again, and ultimately, all nodes in the entire cross-domain cluster can visually display that they have recovered.

[0150] For example, if a node in the "SLAVE_ACCESS_LIST" (i.e., the slave node access list) fails, the other surviving nodes in the "SLAVE_ACCESS_LIST" can still continue to perform cross-domain tasks. The primary master node, mentioned above, is responsible for maintaining RPC heartbeat communication with the "SLAVE_ACCESS_LIST" (i.e., slave node access list) of each secondary and lower-level data lake node. When an abnormal heartbeat is detected in the failed node, the event is reported to the server of the distributed coordination service component, the status of the failed node in the backup information is set to an unreachable state, and this change is broadcast to the entire cluster via RPC. After the failed node recovers, the above process can be automatically triggered again, and ultimately, all nodes in the entire cross-domain cluster can visually demonstrate that they have recovered.

[0151] For example, in the event of a cluster master node failure, the cluster, acting as a client of the distributed coordination service component, receives this notification and, through the distributed coordination service component, elects a new master node from the currently surviving nodes. The new master node can then retrieve a backup of the cross-domain cluster management information from the distributed coordination service component and cache it. This allows cross-domain task collaboration to resume. The master node switchover time is very short, but if new cross-domain task scheduling requires cross-domain cluster management information during the switchover process, it can retry retrieving the information within a certain timeframe, and generally succeeds. However, if the cached information of the new master node is not retrieved after a timeout, it indicates a network failure requiring further investigation. Once the failed master node recovers, a master node re-election operation is not triggered, and normal operation continues.

[0152] For example, in the event of a failure in the distributed coordination service component of a primary node, since the cluster management information within the cluster still relies on the distributed coordination service component, the cluster, acting as a client of the distributed coordination service component, will stop sending cross-domain heartbeats upon receiving this message. The "MASTER_ACCESS_LIST" (i.e., the master node access list) will also return a failure message upon receiving heartbeats and registration information from secondary and lower-level nodes. At this point, the "MASTER_ACCESS_LIST" is offline for secondary and lower-level nodes, and all secondary and lower-level nodes that have lost their heartbeat connection need to be set to an offline state, and the entire cross-domain platform will cease operation.

[0153] Additionally, regarding the recovery of the distributed coordination service component cluster on the primary nodes, the cluster receives a recovery message from the distributed coordination service component server. Because various operations may have occurred during the outage, all secondary and lower-level nodes that were offline will re-register and re-synchronize data and metadata. This process, similar to the deployment process, is automatically triggered and requires no manual intervention.

[0154] For example, in the event of a cluster failure of the distributed coordination service component at the second-level and lower-level nodes, each node in the cluster, acting as a client of the distributed coordination service component, upon receiving this message, will no longer send cross-domain heartbeats from the cluster's master node. The "SLAVE_ACCESS_LIST" (i.e., the slave node access list) will also return a failure message upon receiving heartbeat information from the second-level and lower-level nodes. In other words, the cluster is considered offline and cannot participate in cross-domain tasks.

[0155] In addition, in a distributed data lake system, the real edge data lake (i.e., entity database) at the end of the last-level data lake node can be driven by a predefined data model to achieve edge computing and micro-statistics for the purpose of data self-cleaning and fusion. First, it ensures the correctness and availability of the data itself; second, it submits fused micro-statistical data on demand.

[0156] It's important to note that due to the inherent uncertainty of the underlying data in a data lake (such as internet data), and the potential for unsustainable data provision at its nodes, if a distributed data lake system includes edge data lakes (i.e., entity databases), data loss will occur if the front-end database of these edge nodes is shut down. If a large number of edge nodes shut down, the distributed data lake will become ineffective. To address this issue, a secondary regional database based on business logic (deployed on secondary data lake nodes) is considered. Driven by a unified data model, these secondary data lake nodes will aggregate high-quality, raw, or micro-statistical data within their respective regions. This aggregated data can be seamlessly integrated to support big data analytics within the data lake.

[0157] For example, by constructing predefined data models such as entity models, metadata models, data mapping models, data element models, data quality verification models, data version models, resource catalog models, data service models, and secure access models, data fusion can be driven to standardize the content, format, governance, retrieval, storage, and utilization of data.

[0158] For the data in the secondary data lake nodes, a virtualized data model is established to achieve ETL-free and decentralized processing, and persistent processing is performed.

[0159] Among them, the virtualized data model focuses on the valuable data that the primary database is concerned with to form data entity specifications. The secondary edge database needs to select related data content for abstraction based on the data entity model, and at the same time perform pattern matching and mapping on the data according to the data mapping model. The mapping matching logic includes: acquiring heterogeneous entity data, model matching and mapping, data format standardization, matching temporary storage, data cleaning and transformation, and persistent data storage.

[0160] In this way, it makes up for the shortcomings of traditional data warehouses that require a fixed theme and strong structured storage, and can store untitled, multimodal raw data.

[0161] In addition, by using cloud service technology to access the distributed edge database, the problems of data latency and continuous high bandwidth consumption in traditional ETL are eliminated.

[0162] In addition, by strengthening the global data indexing network in the primary database (deployed on the primary data lake node), reducing the need for physical data storage, and building an integrated distributed data lake driven by a logical model, the problem of the "data swamp" crisis existing in centralized data lakes can be solved.

[0163] Specifically, for secondary data lake nodes, the principle of data mapping is as follows: for the original data of the edge database, relevant data content is selected and abstracted according to the predefined data entity model, and then the data is pattern matched and mapped according to the predefined data mapping model.

[0164] The specific mapping and matching logic is as follows: Assume the edge entity database is A, and the standard database (deployed on the secondary data lake node) of the distributed data lake system is B; Database A contains a data set C, A={C1,C2,…,C…} t}; Database B contains a data set D, B={D1,D2,…,D…} r}; In database A, C i It can be represented as a set of column vectors: C i={P1,P2,…,P X}; Database A contains t tables, each with x fields, forming a data matrix: ; In database B, D i It can be represented as a set of column vectors: D i ={q1,q2,…,q y}; Database B contains r tables, each with y fields, forming a data matrix: ; After the data is aggregated into the standard database B of the secondary regional database, data records are generated for D. i If there are y fields and s records, then the value of each record is... Then D i Constructing a data matrix: ; In database B, based on the predefined data entity model and data mapping model, the pattern matching rules from database A to database B are generated: R={R1,R2,…,R...} m Under the action of pattern matching rule R, for different C i Integrate and generate D i This results in database B.

[0165] The pattern matching rule R includes a three-dimensional matrix plane constructed from the following: data source definition, target database definition, data source table definition, data source data domain definition, target database table definition, target data domain definition, data update mechanism definition, timestamp definition, data update condition combination definition, submitted data range definition, and data audience range definition.

[0166] Based on the pattern matching rules described above, the data of each secondary data lake node is mapped to obtain different data lakes: B1 to B... L Thus, the data matrix K of the secondary data lake nodes is obtained: ; Subsequently, some data in the secondary regional database was standardized according to the SQL standard data specification, and D was set. i Generate data definition rules H=f(D), (the main contents of standardization include field names, field types, field lengths, field definition descriptions, and field business meaning descriptions, etc.), and obtain the standardization matrix. : ; Then, according to the standardized matrix The data in the secondary data lake nodes is processed and cleaned to form fused data that conforms to standards and specifications. Next, the fused state data that conforms to the standard specifications is persisted, and the persisted data is stored in the relevant database.

[0167] In particular, to optimize the execution of query requests, embodiments of this application can model the data lake table according to a standard relational model: Given a set of Mdomains D = {D1, D2, ..., DM}, a table “Tis” is defined as a subset of the Cartesian product D1 × D2 × ... × DM. Each domain D ∈ D has an associated column name C ∈ L, where “Lis” is the set of all column names. TIS is a set of tuples {T1, T2, ..., T|T|}, where each tuple Tis is a pair {(CC∶V)|LV ∈ D, M, ∈ {1, ...,}}. Tis is stored in cloud object storage as a collection of files, represented as := {F1, F2, ..., F||}. Fis is a partition of T, representing ≠J∶FT,∈ <s:2>=T,F∩FJ= .

[0168] The new data lake query Qas definition is given: Definition 1 (Data Lake Query Optimization): A data lake query Qas is defined, where a standard SQL query on the table Tand is represented as Qas< <q>The predicate in the where clause of >. Suppose Qis is given in conjunctive normal form (CNF). If tuple T comes from file F ∈ Fsatisfies, then use S( <s:1>Q, T) represent.

[0169] Definition 2 (Query Coverage): Given a data lake query Q, XQF covers Q. F∈\X, T∈F, S(T,).

[0170] Inference 1: According to Definition 2, for some data lake query Q, when X satisfies as defined in Definition 2, X covers Q (i.e., X covers all files that need to satisfy Q), and uses... <s:2>OV(X, Q) represents this. Such a cover set of Xa is called Q. For any Q, it holds that... <s:2>OV( <e:1>Q).

[0171] Definition 3 (Query Tight Coverage): Given a data lake query Q, XXFtightly covers Q. OV(X, Q) T∈F, S∈Q, T(,) When some data lake query Q has X satisfies defined as 3, it is said that Xtightly covers Q (meaning X contains all the files required to satisfy Q and T), and T <s:2>OV(X, Q) represents a set of Q-closely covering objects. Such a set is called T(Q).

[0172] Definition 4 (Denseness): Given a data lake query Q, for any covering set XofQ, the denseness of Xis is defined as:

[0173] Furthermore, when executing cross-domain query requests in a distributed data lake system, traditional methods read all files. Since F covers any Q, the coverage is not rigorous enough, resulting in poor query performance.

[0174] The embodiments of this application find some covering sets, and the covering set achieves the shortest total query execution time (i.e., finding the covering set X). such The sum of its search costs X and (The size is the smallest). Focusing on the filtering conditions of the query, examine each predicate clause. For each possible coverage execution plan, obtain the expected cost and result of each plan through statistics, caching, ML models, etc. The intersection of the coverage sets of any subset of the query clauses is the coverage set of the original query. Based on this observation, the optimization scheme consists of the following two steps: Step (1): Given a query and its estimate, find a subset of terms and the corresponding data lake scheduling execution plan, where the sum of the intersection of the estimated cost and the data execution time of a subset of terms is minimized.

[0175] Step (2): Execute each coverage plan, insert the results of each coverage plan, and execute the original query conditions only on the files in the intersection.

[0176] Step (1) represents an optimization problem, and a heuristic algorithm is used to handle the hardness of the problem. The result estimation scheme is based on conventional data lake selection size estimation, which can estimate the number of records that satisfy the predicate based on statistical information. Once the estimated number of records R for a given clause is obtained, R is randomly thrown into the box as a ball, and the corresponding number of files is estimated as the number of non-empty boxes.

[0177] Step (2) of the coverage plan execution is based on the indexing work in the cloud data lake. A prototype of the solution was built to achieve online retrieval. The benchmark test used a TPC-H dataset with a scaling factor of 1TB. After optimizing the execution using query requests, a query performance improvement of more than 10 times was achieved.

[0178] In practical applications, as data volume continues to grow, new technologies are constantly being developed to adapt to the increasing data traffic. Cloud data lakes are one of the most successful solutions for addressing the challenges of big data, offering excellent scalability, availability, and cost-effectiveness, but they perform poorly in interactive queries. This application analyzes the problem of poor interactive query performance in cloud data lakes and identifies the main obstacles to achieving better performance. This problem is formally defined by introducing the new concept of query (tight) cover sets, finding the optimal cover set for a given query, and proposing a heuristic algorithm to handle its tightness. This application is based on the ideas of relational databases related to indexing and statistical management, adapted to a cloud data lake architecture. Its main idea is that storage resources are generally less expensive than computing resources, and improving query performance can be demonstrated by adding more storage.

[0179] In practical applications, taking the integrated construction of a digital government data warehouse in a certain province as an example, the optimization of the digital government data warehouse during project implementation utilizes virtualized data model-driven technology. The specific performance improvements of its distributed data lake system structure are as follows: First, selective physical data movement is performed, a hierarchical storage model is adopted, a virtual data view is established, and a network is established between databases of nodes. Physical data movement is only performed as needed. The central data lake (first-level data lake node) synchronizes data with a limited number of second-level regional data lakes (second-level data lake nodes) as needed, which greatly reduces the amount of synchronization tasks and data movement costs.

[0180] Second, the central database deployed on the primary data lake nodes stores only master data and governed, merged data. The enterprise's raw data will be stored on the enterprise side (edge ​​database) and accessed on demand. The central database has a data storage capacity in the GB level, with an annual storage capacity of 200GB × 52 (data updated weekly) × 3 (replicas) = ​​10.2TB, significantly reducing storage costs.

[0181] Third, the central database accesses secondary regional databases deployed on secondary data lake nodes via data routing. Since edge databases report data through cloud services, data latency is mitigated, allowing for the extraction of fresh data for analysis and significantly improving the timeliness of data analysis. By integrating edge computing, regional database construction, big data analysis and mining, and data routing technologies, integrated and interconnected lakehouse data is achieved. This application also provides a computer-readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described method embodiments and achieve the same technical effects. To avoid repetition, these will not be described again here.

[0182] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0183] This application also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0184] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0185] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0187] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application. < / q>

Claims

1. A distributed data lake system, characterized in that, The system includes a primary data lake node and at least one secondary data lake node, and further includes: A global metadata management component, deployed on the primary data lake node, is used to perform hierarchical and unified management of metadata for the primary data lake node and at least one secondary data lake node. The primary data lake node stores a global metadata view and uniformly manages the creation, modification, and deletion of public metadata. The secondary data lake node is used to manage local private metadata. The distributed collaborative management network consists of collaborative agent modules deployed in each secondary data lake node; wherein, the collaborative agent module is used to elect a master node within its own node cluster; register the status information of its own node with the primary data lake node; and obtain the status information of other secondary data lake nodes from the primary data lake node; The data mapping engine, deployed on the secondary data lake node, is used to map and process raw data from the edge database into fused data that conforms to global specifications, based on a predefined data model.

2. The system according to claim 1, characterized in that, The primary data lake nodes are used to store the fused data and global metadata indexes after governance. The secondary data lake node is used to store raw data from the edge database and fused data obtained by mapping the raw data. The system is used to perform cross-node data transfer or computation tasks based on the logical location of the data determined by the global metadata management component and the real-time node status provided by the distributed collaborative management network.

3. The system according to claim 1 or 2, characterized in that, The distributed collaborative management network is specifically used to: manage the cluster status information of the secondary data lake nodes; Specifically, after the secondary data lake node is started, a master control node is determined through an election mechanism. The master control node sends the registration information of the specified access node list to the primary data lake node. After verifying the registration information, the primary data lake node triggers the synchronization of data and metadata with the secondary data lake node and caches the cluster status information of the secondary data lake node. After the access node list information of the secondary data lake node is successfully registered, the primary data lake node caches the access node list information of the secondary data lake node and synchronizes it to other registered secondary data lake nodes.

4. The system according to claim 3, characterized in that, The distributed collaborative management network is also used for: The master control node maintains heartbeat communication with the access node list of each secondary data lake node to monitor node status; When a node failure or recovery is detected, the state change information is notified to other secondary data lake nodes via remote procedure call, updating the cluster state information cached by the other secondary data lake nodes.

5. The system according to claim 1, characterized in that, The system also includes a distributed coordination service component; The distributed coordination service component is deployed on the primary data lake node and is used to re-elect a new primary control node in the event of a failure of the primary control node on the secondary data lake node. The new master node is used to obtain the backed-up cluster status information from the distributed coordination service component and re-cache it.

6. The system according to any one of claims 1-5, characterized in that, The data mapping engine is used to: determine pattern matching rules from the edge database to the secondary data lake node based on a predefined data entity model and data mapping model; map and aggregate the raw data from the edge database to the secondary data lake node based on the pattern matching rules; standardize and clean the raw data according to the standardization specifications to obtain fused data that conforms to the global specifications, and persistently store the fused data.

7. The system according to claim 6, characterized in that, The data mapping engine is specifically used for: mapping and aggregating the original data to the secondary data lake node based on the pattern matching rules; processing the data in the secondary data lake node into a data matrix; and standardizing the data matrix according to standard data specifications to generate a standardized matrix. The data in the standardized matrix is ​​cleaned to form fused data that conforms to global specifications.

8. The system according to any one of claims 1-5, characterized in that, The system also includes a query optimization execution module; the query optimization execution module is used to receive query requests, determine a coverage set based on the statistical information of query predicates, and obtain the final intersection by executing the subquery plan of the coverage set and taking the intersection of the results. Execute the query request on the data files in the final intersection.

9. The system according to claim 8, characterized in that, The query optimization execution module is specifically used for: parsing the filtering conditions in the query request and decomposing the filtering conditions into multiple logical sub-conditions; for at least one of the multiple logical sub-conditions, determining the set of data files that need to be accessed to satisfy the at least one logical sub-condition based on the statistical information of the data lake system, and obtaining a coverage set; forming candidate execution plans by combining different coverage sets, and selecting the candidate execution plan with the lowest estimated cost to execute the query request.

10. A performance optimization method for a distributed data lake system as described in any one of claims 1-9, characterized in that, include: After starting up and electing a master node, the secondary data lake node registers with the primary data lake node; The primary data lake node verifies registration and triggers data synchronization, and synchronizes and caches cluster status information among all secondary data lake nodes; Based on a predefined data model, the raw data stored in the edge database is mapped and standardized and cleaned to obtain fused data that conforms to global specifications, and the fused data is persistently stored. For a query request, a coverage set is determined based on the statistical information of the query predicate. The final intersection is obtained by executing the subquery plan of the coverage set and taking the intersection of the results. Execute the query request on the data files in the final intersection.

Images