Metadata query method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a metadata query method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of databases. The method comprises the following steps: based on a partition filtering operator of a data query request, scanning a partition directory of the distributed file system to determine at least one specified partition directory; and based on the time filtering operator, scanning metadata of the data file under each specified partition directory, and obtaining at least one piece of matched metadata of the data query request to obtain a query result. According to the method and the device, the appointed partition directory is determined through the partition filtering operator, so that the partition screening of the metadata is realized. A data file list in a specified time interval is determined through a time filtering operator, and screening of time conditions of metadata is achieved. According to the method, high-performance and high-efficiency query of the metadata is realized through the partition filtering operator and the time filtering operator, and the matched metadata can be quickly obtained.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a metadata query method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] The Hadoop Distributed File System (HDFS) is the foundation of data storage management in distributed computing. Its high fault tolerance, reliability, scalability, availability, and throughput provide fault-tolerant storage for large datasets, greatly facilitating the processing of massive datasets. HDFS relaxes some POSIX constraints to enable streaming file system data access. HDFS applications require a write-once, read-many file access model. The assumption that a file should not be modified after creation, writing, and closing simplifies data consistency issues and enables high-throughput data access. HDFS employs a master / slave architecture, and from the end-user's perspective, it operates like a traditional file system, performing create-read-update-delete (CRUD) operations on files via directory paths.

[0003] In big data scenarios, the daily volume of newly added data is large, the data retention period is long, and the number of partitions can reach millions. Current technologies have problems. When executing computational tasks, it is necessary to first scan and read the full metadata of HDFS (including all directories and files), form a view of the metadata in memory, and then traverse which files need to be read based on partition filtering conditions. The problem with this approach is that when the number of partitions and files is large (e.g., hundreds of thousands or even millions), a full scan of HDFS metadata is very slow, and the metadata consumes a large amount of memory on the computing nodes, potentially even causing memory overflow. Summary of the Invention

[0004] This application provides a metadata query method, apparatus, electronic device, storage medium, and program product to address the shortcomings of existing technologies where full scanning of HDFS metadata consumes a large amount of computing node memory, thereby reducing the amount of HDFS metadata scanned.

[0005] Firstly, this application provides a metadata query method, including: Receive data query requests, which include partition filtering operators and time filtering operators; Based on the partition filtering operator, scan the partition directories of the distributed file system to identify at least one specified partition directory; Based on the time-based filtering operator, the metadata of data files in each specified partition directory is scanned to obtain at least one matching metadata for the data query request, thereby obtaining the query results.

[0006] In one embodiment, the distributed file system includes a metadata index tree, which is constructed based on the following: The root node is constructed based on the overall attribute values ​​of the data table; Intermediate nodes are constructed based on at least one form of partitioning metric from the metadata in the data table; Based on the partition values, storage time information, and storage path of the metadata in the data table, construct leaf nodes; A metadata index tree is constructed based on the root node, intermediate nodes, and leaf nodes, forming a partitioned directory.

[0007] In one embodiment, based on a partition filtering operator, the partition directories of the distributed file system are scanned to determine at least one specified partition directory, including: Based on the specified partition value of the partition filtering operator, traverse the partition values ​​of the root node and the intermediate nodes under the root node, and filter the intermediate nodes that meet the specified partition value. Get the directory of the specified partition based on the intermediate nodes that satisfy the specified partition value.

[0008] In one embodiment, the query results include a list of data files. Based on a time-based filtering operator, the metadata of the data files in each specified partition directory is scanned to obtain at least one matching metadata from the data query request, thereby obtaining the query results, including: Based on the specified time interval of the time filtering operator, it iterates through the storage time information of metadata in the leaf nodes under the specified partition directory, and uses the metadata that meets the specified time interval as the matching metadata. Merge the storage paths of the matching metadata to merge the matching metadata and obtain at least one target data file; Based on each target data file, obtain a list of data files.

[0009] In one embodiment, based on a time-based filtering operator, the metadata of data files in each specified partition directory is scanned to obtain at least one matching metadata for the data query request. After obtaining the query results, the method further includes: When the data file list includes multiple data pages, construct the association information between the data pages and obtain the associated data pages; The associated data pages are sent to the data query terminal so that the data query terminal can continuously read the data pages.

[0010] In one embodiment, the query results also include storage characteristics of the matching metadata. After obtaining the query results, the results also include: Receive data scanning instructions sent by the computing node. The data scanning instructions include a scanning strategy, which is determined based on the storage characteristics of the matching metadata. Based on the scanning strategy, scan the matching metadata and obtain the status report of the data block matching the metadata.

[0011] Secondly, this application also provides a metadata query device, comprising: The receiving module is used to receive data query requests, which include partition filtering operators and time filtering operators. The partition filtering module is used to scan the partition directories of the distributed file system based on partition filtering operators to identify at least one specified partition directory. The time filtering module is used to scan the metadata of data files in each specified partition directory based on time filtering operators, obtain at least one matching metadata of the data query request, and obtain the query results.

[0012] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the metadata query methods described above.

[0013] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the metadata query methods described above.

[0014] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the metadata query methods described above.

[0015] This application provides a metadata query method, apparatus, electronic device, storage medium, and program product that receives a data query request, which includes a partition filtering operator and a time filtering operator. Based on the partition filtering operator, it scans the partition directories of a distributed file system to determine at least one specified partition directory. Based on the time filtering operator, it scans the metadata of data files in each specified partition directory to obtain at least one matching metadata of the data query request, thereby obtaining the query result. This application achieves partition filtering of metadata by determining the specified partition directory through the partition filtering operator. It achieves time filtering of metadata by determining the list of data files within a specified time interval through the time filtering operator. This application achieves high-performance and high-efficiency metadata querying through partition and time filtering operators, enabling rapid acquisition of matching metadata. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts illustrating the metadata query method provided in this application.

[0018] Figure 2 This is the second flowchart illustrating the metadata query method provided in this application.

[0019] Figure 3 This is a schematic diagram of the metadata query device provided in this application.

[0020] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] An HDFS cluster consists of a NameNode and several DataNodes. The NameNode manages the file system's metadata, while the DataNodes store the actual data. Clients access the file system by interacting with both the NameNode and DataNodes. Clients contact the NameNode to retrieve file metadata, while the actual file input / output (I / O) operations are performed directly with the DataNodes. The NameNode and DataNodes can run on ordinary commercial machines; one NameNode instance can run on a single machine, while the other machines in the cluster can each run a DataNode instance, or multiple DataNodes can run on a single machine.

[0023] The existing HDFS metadata management process has the following core shortcomings.

[0024] (1) Uncontrolled memory resource consumption. The essence of this problem is that the compute nodes need to fully load the partition metadata (such as the path, file size, block location, etc. corresponding to millions of partitions), and the memory consumption of a single task can reach GB to TB. The resulting derivative risks include: frequent compute node crashes, especially in memory-constrained containerized environments (such as K8s containers); and squeezing the memory quota of actual computing tasks, reducing data processing efficiency.

[0025] (2) Network transmission bottlenecks are exacerbated due to the defects of the storage-compute separation architecture and the cost amplification effect. Specific defects of the storage-compute separation architecture include: metadata files are transmitted over a 10GE network (theoretical peak 1.25GB / s), and when the metadata of a million partitions is approximately 50–100GB, a single load takes 40–80 seconds; and thousands of tasks repeatedly pull the same metadata every day, resulting in network bandwidth utilization exceeding 60% (actual test data). The cost amplification effect is reflected in the high traffic costs generated by cross-Availability Zone (AZ) / Region transmission in the cloud environment.

[0026] (3) The metadata access mode is rigid. This is because if the full loading anti-mode is adopted, even if only one partition needs to be accessed (e.g., the partition with the time of January 1, 2023), all historical partition metadata still needs to be loaded, and the effective data ratio is <0.001%. Moreover, due to the drag of cold data, historical partitions (e.g., data from 7 years ago) are still forcibly loaded, which violates the principle of data locality.

[0027] (4) Lack of scalability and elasticity, including: single-point bottleneck of metadata files and failure of dynamic scaling, which makes the centralized metadata file a single point of performance, with concurrent tasks competing for read and write locks; the increase in the number of partitions leads to linear expansion of file size, exceeding the small file processing capacity of HDFS (NameNode memory pressure). The failure of dynamic scaling means that when the computing cluster is expanded, new nodes need to fully synchronize metadata, with latency reaching hours.

[0028] (5) Insufficient real-time guarantee, manifested in metadata update delays and the inability to support incremental updates. The generation of metadata files on the writing end and the writing of data to HDFS are non-atomic operations, resulting in a minute-level inconsistency window; computing tasks may read expired metadata, causing FileNotFoundException. In the absence of incremental updates, the entire metadata file needs to be rewritten for each change, significantly amplifying input and output (I / O).

[0029] (6) Poor cross-engine compatibility is reflected in the binding to specific computing frameworks and the fragmentation of metadata semantics. Custom metadata file formats (such as Parquet / JSON) need to be adapted to different engines (Spark / Flink / Presto), which greatly increases the maintenance cost; and the metadata semantics are fragmented, so native metadata services such as Hive Metastore (Hive metadata storage) and Iceberg table format cannot be reused.

[0030] In the existing metadata query process, actual tests show that 60% of the computation cycle is consumed in the metadata preparation stage. When the number of partitions exceeds tens of millions, the existing solution becomes completely unusable. At the same time, the operation and maintenance complexity is high, requiring manual tuning of parameters (such as spark.driver.memory=20G) to mask the design flaws.

[0031] Existing metadata queries only optimize the metadata generation path (avoiding full scans of HDFS), but fail to address fundamental issues such as storage format (full redundancy), transmission protocol (no streaming filtering), and access interface (lack of predicate pushdown). Therefore, it is necessary to build a lightweight indexing service (such as Bloom filters + partition pruning), columnar metadata storage (optimized row columnar (ORC) format for on-demand reading), and a distributed caching layer (Alluxio for caching hot metadata) to decouple metadata from computing resources at the architectural level.

[0032] The following is combined with Figures 1-4 This application describes the metadata query method, apparatus, and electronic device.

[0033] Figure 1 This is one of the flowcharts illustrating the metadata query method provided in this application, such as... Figure 1 As shown, the metadata query method is applied to a distributed file system and includes steps S100 to S200, each of which is detailed below.

[0034] S100: Receives data query requests, which include partition filtering operators and time filtering operators.

[0035] The execution entity in this application embodiment includes a distributed file system. The distributed file system includes the Hadoop Distributed File System (HDFS). This application embodiment uses HDFS as an example for illustration.

[0036] like Figure 2 As shown, the compute node sends a user query request for lakeware metadata to the data query client. The data query client then sends a data query request to HDFS based on the user query request.

[0037] The data query end includes a lake warehouse management component. This component includes a data lake management framework (Hudi). The lake warehouse management component (e.g., Hudi) is used to persist written data files to the HDFS directory structure in an open format, with data files placed in different directories according to defined partitions. Datanodes store HDFS data as files in their local file system, unaware of information about the HDFS files, storing each HDFS data block in a separate file on the local file system. This application builds a unified lake warehouse architecture on top of HDFS, with metadata including lake warehouse metadata. Based on technologies such as batch processing (Spark), stream processing (Flink), and tabular processing (Iceberg), this unified lake warehouse architecture combines the advantages of data lakes and data warehouses, directly implementing the strict data management required in data warehouses onto low-cost distributed storage. The unified lake warehouse architecture is characterized by the adoption of Atomicity-Consistency-Isolation-Durability (ACID) principles, schema management, separation of storage and computation, and support for multiple computing engines and file formats.

[0038] Define the data query client and protocol. For example, define the port type, endpoint address, and authentication mechanism of the data query client. For instance, design based on the Representational State Transfer Application Programming Interface (RESTful API), using HTTP / HTTPS protocols to provide a standardized data query client port. Define the authentication mechanism for the data query client. The authentication mechanism includes verifying the identity of the compute node through API keys (e.g., X-API-Key request headers) or OAuth 2.0 tokens.

[0039] The compute nodes send user query requests for lakeware metadata to the data query client. User query requests include user-input filtering conditions. Compute nodes include batch processing and stream processing (Spark+Flink) nodes. User query requests can use Structured Query Language (SQL) commands. User query requests include data grouping query requests and data grouping filtering requests. Data grouping query requests are specific to SQL; SQL categorizes retrieved tuples according to a certain condition, grouping tuples with the same condition value into a group or set, and simultaneously processing aggregation operations on multiple groups or sets. Grouping filtering, based on grouping, retains sets (groups) that meet the conditions and discards those that do not. User query requests include filtering conditions and a list of metadata fields to be returned. Optionally, filtering conditions include the target database name, table type, and query time. The list of metadata fields to be returned includes the table name, metadata schema version, and the storage path of the matching metadata.

[0040] Optionally, after receiving a user's query request, the data query client sends a response to the compute node. The response includes a success response and an error response. The error response carries the reason for the error, such as an invalid filter condition, a spelling error, or the existence of an invalid field.

[0041] The data query client extracts the specified partition value and time range based on the filtering conditions in the user's query request. It then constructs a partition filtering operator based on the target partition value and a time filtering operator based on the target time range. Finally, based on the partition filtering operator, the time filtering operator, and the list of metadata fields to be returned, the data query client constructs a data query request and sends it to HDFS (HDFS Distributed File System).

[0042] Optionally, an HDFS interface is designed to enable communication and interaction between HDFS and the data query client. The HDFS interface uses remote procedure call (gRPC) or an optimized version of the HTTP / 2 protocol to support high-concurrency, low-latency requests.

[0043] S200: Based on the partition filtering operator, scan the partition directories of the distributed file system to identify at least one specified partition directory.

[0044] HDFS uses partition filtering operators in data query requests to scan the partition directories of the distributed file system and identify those that meet certain criteria. The specified partition directory is then formed by combining all matching directories.

[0045] S300: Based on time filtering operators, scan the metadata of data files in each specified partition directory to obtain at least one matching metadata of the data query request in order to obtain the query results.

[0046] HDFS uses time-filtering operators in data query requests to scan metadata in data files under a specified partition directory and obtain matching metadata within a specified time interval.

[0047] Based on the matching metadata, obtain a list of data files (query results). Optionally, generate a target data file based on each matching metadata, and obtain a list of data files based on all target data files.

[0048] Metadata includes lakewarehousing metadata. Lakewarehousing metadata is stored using a hierarchical directory structure. For example, the storage path for lakewarehousing metadata includes data lake-database name-table name-partition value 1 and partition value 2. Metadata is data that describes data; it's an explanation or description of other data, used to provide information about the structure, content, path origin, etc., of a particular dataset.

[0049] The list of data files includes the storage path for matching metadata, which can be used to query the matching metadata.

[0050] The metadata query method provided in this application receives a data query request, which includes a partition filtering operator and a time filtering operator. Based on the partition filtering operator, it scans the partition directories of the distributed file system to determine at least one specified partition directory. Based on the time filtering operator, it scans the metadata of data files under each specified partition directory to obtain at least one matching metadata of the data query request, thereby obtaining the query result. This application uses the partition filtering operator to determine the specified partition directory, thus achieving partition filtering of metadata. It uses the time filtering operator to determine the list of data files within a specified time interval, thus achieving time filtering of metadata. This application achieves high-performance and high-efficiency metadata querying through partition filtering and time filtering operators, enabling rapid acquisition of matching metadata.

[0051] Based on the above embodiments, the distributed file system includes a metadata index tree, which is constructed in the following manner: The root node is constructed based on the overall attribute values ​​of the data table; Intermediate nodes are constructed based on at least one form of partitioning metric from the metadata in the data table; Based on the partition values, storage time information, and storage path of the metadata in the data table, construct leaf nodes; A metadata index tree is constructed based on the root node, intermediate nodes, and leaf nodes, forming a partitioned directory.

[0052] The overall attribute values ​​of a data table include the hash value of the table name or the location of the primary index. Root node shards are mapped to physical storage nodes.

[0053] At least one form of partitioning metric includes time-based metrics, regional metrics, file format, or file schema (file_format). A time-based partitioning metric could be used, for example, data collected in each year (time-based metric) as a partition. A regional partitioning metric could be used, for example, data collected in Asia as one partition and data collected in Europe as another. Intermediate nodes consist of a B+ tree index built according to the partitioning metric (table attribute). Leaf nodes store specific metadata, including the partition value, storage time information, and storage path.

[0054] The metadata index tree carries data table attributes. It includes a tree-structured partition directory, along with the metadata index (storage path) of each partition directory and the storage time information for the metadata. The storage time information is stored in a tree structure according to year, month, day, and time point. For example, time point 1 (time1), time point 2 (time2), time point 3 (time3), time point 4 (time4), and time point 5 (time5) are stored from top to bottom in various tree-structured partition folders, thus forming hierarchical and indexable timestamp information.

[0055] The root node is constructed based on the overall attribute values ​​of the data table. For example, the root node is constructed based on the hash value of the data table name, and the root node is sharded and mapped to physical storage nodes.

[0056] Intermediate nodes are constructed based on the partitioning metrics of the metadata in the data table. A B+ tree index is then built based on the partitioning metrics of the metadata in the data table to obtain the intermediate nodes. Based on the root node, intermediate nodes, and leaf nodes, a metadata index tree is constructed, and the root node, intermediate nodes, and leaf nodes form the partition directory.

[0057] The following code describes the metadata index tree.

[0058] Root, (Table Hash) │ ├─Partition Key: "date" (B+ Tree) │├─ Leaf: date=20231001 → [S3 path 1, time range: 20231001-20231001] │└─ Leaf: date=20231002 → [S3 path 2, time range: 20231002-20231002] │ └─Partition Key: "region" (B+ Tree) ├─ Leaf: region=asia → [S3 Path 3, Time Range: 20231001-20231005] └─ Leaf: region=europe → [S3 path 4, time range: 20231001-20231003].

[0059] In this configuration, "Partition Key: "date" (B+ Tree)" indicates that time is used as the partition criterion, employing a B+ tree data structure with Root as the root node and Table Hash as the table's hash value. "Leaf: date=20231001" indicates that the leaf node's partition value is October 1, 2023. "S3 path 1", "S3 path 2", "S3 path 3", and "S3 path 4" are all storage paths within the leaf nodes. "Time range: 20231001-20231001", "Time range: 20231002-20231002", "Time range: 20231001-20231005", and "Time range: 20231001-20231003" are all time information stored within the leaf nodes. "Partition Key: "region" (B+ Tree)" indicates that a region is used as the partition criterion, employing a B+ tree data structure. "region=asia" means the leaf node's partition value is Asia, and "Leaf: region=europe" means the leaf node's partition value is Europe.

[0060] The metadata index tree uses a hierarchical directory structure to store metadata, following Hive-style partition naming conventions. The specific storage path for a single piece of metadata is as follows.

[0061] s3: / / lakehouse / sales_db / orders / region=asia / date=20231001 / .

[0062] This code indicates that the matching metadata is stored in an S3 bucket named "lakehouse". The matching metadata belongs to a table named "order" in a database named "sales_db". The matching metadata is partitioned by region, with the partition value being "asia". The storage time information for the matching metadata is October 1, 2023.

[0063] This application constructs a metadata index tree by establishing root nodes, intermediate nodes, and leaf nodes. This supports complex queries based on partition and time filtering, avoiding a full scan of metadata and improving metadata retrieval efficiency. The metadata index tree can quickly remove irrelevant data fragments based on partition and time filtering, thus achieving dynamic pruning. This application supports at least one form of partitioning metric (e.g., time metric, region metric, or file format), enabling the metadata index tree to automatically merge query conditions, thereby improving the scalability of data query requests.

[0064] Based on the above embodiments, scanning the partition directories of the distributed file system using a partition filtering operator to determine at least one specified partition directory includes the following steps: Based on the specified partition value of the partition filtering operator, traverse the partition values ​​of the root node and the intermediate nodes under the root node, and filter the intermediate nodes that meet the specified partition value. Get the directory of the specified partition based on the partition values ​​of the intermediate nodes that satisfy the specified partition values.

[0065] Optionally, intermediate nodes store partition values. Partition values ​​are the specific values ​​taken under a partition indicator. For example, when the partition indicator is a region, the partition values ​​might be Asia, Europe, etc. The range of partition values ​​for intermediate nodes is greater than the range of partition values ​​for leaf nodes. For example, if the partition value for an intermediate node is Asia, then the partition values ​​for leaf nodes might be specific countries within Asia, such as country A or country B within Asia.

[0066] The partition filtering operator includes a specified partition value. It iterates through each root node and all intermediate nodes under the root node based on the specified partition value, filtering out intermediate nodes that satisfy the specified partition value. Based on the specified partition value, it locates the B+ tree branch under the root node. For example, if the specified partition value is Asia, it locates the intermediate node (B+ tree branch) with partition value Asia.

[0067] Retrieve the specified partition directory based on intermediate nodes that satisfy the specified partition values. Optionally, the specified partition directory may include multiple partition values ​​that satisfy the specified partition values.

[0068] Get the directory of the specified partition based on the intermediate nodes that satisfy the specified partition value.

[0069] Optionally, the partitioning filter operator carries credentials to ensure secure data access. Credentials include access keys for Amazon Web Services (AWS).

[0070] For example, the partition filtering operator is: "Please visit the dataset named 'sales_db' in the data lake of the S3 bucket in Region A, with the data table named 'orders', and execute an SQL query to find all records with the partition value 'asia'. To authorize this access, an AWS access key is provided here."

[0071] This application enables fast retrieval of branches of the metadata index tree by specifying partition values ​​and filtering root and intermediate nodes, which helps improve the efficiency of metadata retrieval.

[0072] Based on the above embodiments, the query results include a list of data files. Using a time-based filtering operator, the metadata of data files in each specified partition directory is scanned to obtain at least one matching metadata entry for the data query request, thus obtaining the query results. This includes the following steps: Based on the specified time interval of the time filtering operator, it iterates through the storage time information of metadata in the leaf nodes under the specified partition directory, and uses the metadata that meets the specified time interval as the matching metadata. Merge the storage paths of the matching metadata to merge the matching metadata and obtain at least one target data file; Based on each of the target data files, obtain the list of data files.

[0073] Based on a specified time interval, the storage time information of matching metadata in the leaf nodes of a specified partition directory is filtered to obtain leaf nodes that satisfy the specified time interval. In this case, the storage time information of the leaf nodes in the specified partition directory is November 1, 2023. The specified time interval is November 1, 2023 - November 2, 2023. Therefore, the leaf node corresponding to November 1, 2023, is determined to satisfy the specified time interval.

[0074] Merge the storage paths of metadata in leaf nodes that satisfy a specified time interval to obtain at least one data file. Optionally, merge identical storage paths among the metadata storage paths in leaf nodes that satisfy a specified time interval to obtain multiple target data files. The target data files include not only the storage paths of the matching metadata or the merged storage paths, but also the storage time information, partition value, file size, compression format, etc., of the matching metadata. Obtain a list of data files based on all the target data files.

[0075] This application enables rapid retrieval of specific metadata by filtering leaf nodes based on a specified time interval. The data file list is simplified by merging storage paths.

[0076] Based on the above embodiments, after scanning the metadata of data files in each specified partition directory using a time-based filtering operator to obtain at least one matching metadata for the data query request and then obtaining the query results, the following steps are also included: When the data file list includes multiple data pages, construct the association information between the data pages and obtain the associated data pages; The associated data pages are sent to the data query terminal so that the data query terminal can continuously read the data pages.

[0077] After retrieving the list of data files within a specified time interval, the distributed file system sends the list to the data query client. Specifically, the returned data file list format includes JSON Lines. JSON Lines allows the data query client to parse the data as it is received, reducing end-to-end latency and thus providing streaming processing capabilities. The returned data file list may include the metadata storage path, record_count, size, and schema_version fields. The record_count and size fields help compute nodes predict computing resource requirements, enabling accurate measurement. schema_version supports the coexistence of old and new metadata formats, thus achieving version compatibility.

[0078] When a list of data files comprises multiple data pages, the association information between the data pages is constructed, and the associated data pages are retrieved. The Distributed File System (HDFS), when transmitting a list of data files to a data query client, needs to divide the list into multiple data pages to transmit them sequentially. During pagination, the association information between the data pages is constructed, and the associated data pages are retrieved. For example, each data page can be represented as an HTTP response header, carrying pagination control flags (association information).

[0079] The associated data pages are sent to the data query terminal so that the data query terminal can continuously read the data pages.

[0080] The pagination mechanism in this application enables seamless expansion of the sharded reading logic under a big data framework.

[0081] Furthermore, the data query end continuously reads multiple data pages transmitted from the Distributed File System (HDFS) to obtain a list of data files.

[0082] The data query client returns a list of data files to the compute nodes. A transport layer is constructed between the data query client and the compute nodes. For example, a transport layer based on Apache Arrow Flight Remote Procedure Call (RPC) can be built between the data query client and the compute nodes, supporting zero-copy batch transmission of large data sets, with built-in flow control and parallel transmission mechanisms in the protocol.

[0083] During the process of the data query client returning a list of data files to the compute node, Transport Layer Security (TLS) and signatures are enabled to ensure the security of data transmission. Optional TLS protocols include mutual TLS authentication. Signatures include Amazon Web Services Signature Version 4 (AWS SigV4) or AWS SigV6.

[0084] During the process of the data query terminal returning a list of data files to the computing node, an enhanced version of the data file list can be used. This enhanced data file list includes not only the storage path of the matching metadata but also the storage characteristics of the matching metadata. These storage characteristics include the compression algorithm, encoding method, and memory size of the matching metadata.

[0085] Based on the above embodiments, the query results also include the storage characteristics of matching metadata. After obtaining the query results, the following steps are also included: Receive data scanning instructions sent by the computing node. The data scanning instructions include a scanning strategy, which is determined based on the storage characteristics of the matching metadata. Based on the scanning strategy, scan the matching metadata and obtain the status report of the data block matching the metadata.

[0086] After receiving the enhanced list of data files from the data query client, the compute node creates a scanning strategy for the matching metadata based on its storage characteristics. It then creates a data scan instruction based on the scan strategy and the enhanced list of data files. The compute node sends the data scan instruction to the HDFS distributed file system. HDFS starts a Datanode based on the data scan instruction, scans the local matching metadata according to the scan strategy, and obtains a status report of the matching metadata data blocks based on the scan results, thereby ensuring the healthy operation of the entire HDFS system.

[0087] Furthermore, during the matching metadata scanning process, the Datanode, upon startup, confirms and triggers the scan based on the hardware self-test results and the compute node's reception of the matching metadata, using a confirmation signal. For example, the scan will only be initiated after the hardware diagnostics pass and the necessary matching metadata has been successfully received.

[0088] Furthermore, during the scanning process of matching metadata, an emergency scanning mode (preemptive resource scheduling) is configured for newly listed matching metadata to ensure that the first status report is completed within a short time (e.g., 30 seconds).

[0089] Furthermore, a hybrid scanning mode is employed during the metadata matching scan. This hybrid scanning mode combines incremental scanning with full verification. Incremental scanning includes monitoring via kernel file system event notification (inode notify, inotify). Full verification includes cyclic redundancy check (CRC) indexing, such as CRC32. Incremental scanning is achieved by capturing file change events in real time through the Linux kernel's inotify API (Application Programming Interface). During the scan, a baseline verification file is generated periodically to establish a full baseline.

[0090] During or after a scan, the DataNode generates a status report. This report contains information about all scanned data blocks and is then sent to the NameNode. Upon receiving the report, the NameNode updates its global metadata view to understand the data distribution and health status of the entire cluster.

[0091] The metadata query method provided in this application has the following advantages.

[0092] (1) By building a lake warehouse metadata index tree in the storage and overlaying it with a push-down method that supports partition filtering operators, high-performance and high-efficiency querying of lake warehouse metadata under large data volume is achieved, enabling computing nodes to quickly obtain the matching metadata required for computing.

[0093] (2) Supports combined filtering (such as partition filtering and time filtering) and dynamic field projection, improving the flexibility of metadata query. By expanding the key-value pairs of filtering conditions, it is compatible with future added metadata attributes, which can improve the scalability of metadata query. At the same time, by utilizing the metadata index tree, it accelerates the query of time range and partition range, and optimizes the query of metadata.

[0094] (3) By using B+ tree indexes to support O(log n) complexity queries, full scanning of metadata is avoided, thus improving index efficiency. Dynamic pruning is achieved by quickly removing irrelevant data shards through dual filtering of time intervals and partition values. This application supports mixed partitioning (such as regional partitioning metrics and time partitioning metrics), and the index tree automatically merges multi-condition queries to improve scalability.

[0095] (4) The use of JSON Lines format allows the client to receive and parse data simultaneously, reducing end-to-end latency and thus providing streaming processing capabilities. When returning a list of data files, the record_count and size fields help computing nodes predict computing resource requirements, thereby achieving accurate measurement; the schema_version field of metadata supports the coexistence of new and old metadata formats to achieve version compatibility; the pagination mechanism can be connected to the sharding reading logic of big data frameworks such as Spark / Hadoop, thus enabling seamless expansion.

[0096] (5) Metadata carries storage characteristics (e.g., encoding / compression), which can guide the computing engine (or computing nodes) to optimize reading strategies, thereby achieving computing-storage collaboration. The Arrow memory format is natively compatible with computing frameworks such as Spark and Pandas, achieving zero-copy acceleration. Dynamic adjustment of parallelism and compression strategies adapts to diverse infrastructure environments, enabling elastic transmission. End-to-end verification and encryption ensure enterprise-level data security requirements, achieving trusted delivery.

[0097] (6) Hybrid scanning mode can detect critical block changes within 1 second, enabling rapid discovery. A binary protocol based on Protocol Buffers (ProtoBuf) ensures lossless metadata transmission, enabling precise repair. Sharded streaming supports single-node management of millions of blocks, enabling elastic scaling. Vectorized computation and Non-Volatile Memory Express (NVMe) optimization achieve sub-millisecond response times, enabling hardware collaboration. Self-Monitoring Analysis and Reporting Technology (SMART) combined with disk health monitoring enables storage media fault preprocessing, providing self-healing capabilities.

[0098] The metadata query apparatus provided in this application is described below. The metadata query apparatus described below and the metadata query method described above can be referred to in correspondence.

[0099] A metadata query device, comprising: The receiving module 301 is used to receive data query requests, which include partition filtering operators and time filtering operators.

[0100] The partition filtering module 302 is used to scan the partition directories of the distributed file system based on the partition filtering operator to identify at least one specified partition directory.

[0101] The time filtering module 303 is used to scan the metadata of data files in each specified partition directory based on the time filtering operator, obtain at least one matching metadata of the data query request, and obtain the query result.

[0102] The metadata query apparatus provided in this application receives a data query request, which includes a partition filtering operator and a time filtering operator. Based on the partition filtering operator, it scans the partition directories of the distributed file system to determine at least one specified partition directory. Based on the time filtering operator, it scans the metadata of data files under each specified partition directory to obtain at least one matching metadata of the data query request, thereby obtaining the query result. This application achieves partition filtering of metadata by determining the specified partition directory through the partition filtering operator. It achieves time filtering of metadata by determining the list of data files within a specified time interval through the time filtering operator. This application achieves high-performance and high-efficiency metadata query through partition filtering and time filtering operators, enabling rapid acquisition of matching metadata.

[0103] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0104] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions in the memory 330 to execute a metadata query method. This method includes: receiving a data query request, the data query request containing a partition filtering operator and a time filtering operator; scanning the partition directories of the distributed file system based on the partition filtering operator to determine at least one specified partition directory; and scanning the metadata of data files in each specified partition directory based on the time filtering operator to obtain at least one matching metadata of the data query request, thereby obtaining a query result.

[0105] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the metadata query method provided by the above methods. The method includes: receiving a data query request, the data query request including a partition filtering operator and a time filtering operator; scanning the partition directories of a distributed file system based on the partition filtering operator to determine at least one specified partition directory; and scanning the metadata of data files in each specified partition directory based on the time filtering operator to obtain at least one matching metadata of the data query request to obtain a query result.

[0107] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the metadata query method provided by the above methods. The method includes: receiving a data query request, the data query request including a partition filtering operator and a time filtering operator; scanning the partition directories of a distributed file system based on the partition filtering operator to determine at least one specified partition directory; and scanning the metadata of data files under each specified partition directory based on the time filtering operator to obtain at least one matching metadata of the data query request to obtain a query result.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A metadata query method, characterized in that, include: Receive a data query request, which includes a partition filtering operator and a time filtering operator; Based on the partition filtering operator, scan the partition directories of the distributed file system to identify at least one specified partition directory; Based on the time filtering operator, the metadata of data files in each specified partition directory is scanned to obtain at least one matching metadata of the data query request, so as to obtain the query result.

2. The metadata query method according to claim 1, characterized in that, The distributed file system includes a metadata index tree, which is constructed based on the following: The root node is constructed based on the overall attribute values ​​of the data table; Intermediate nodes are constructed based on at least one form of partitioning metric from the metadata in the data table; Based on the partition values, storage time information, and storage paths of the metadata in the data table, leaf nodes are constructed; The metadata index tree is constructed based on the root node, the intermediate nodes, and the leaf nodes, and the root node, the intermediate nodes, and the leaf nodes form the partition directory.

3. The metadata query method according to claim 2, characterized in that, The step of scanning the partition directories of the distributed file system based on the partition filtering operator to determine at least one specified partition directory includes: Based on the specified partition value of the partition filtering operator, traverse the partition values ​​of the root node and the intermediate nodes under the root node, and filter the intermediate nodes that satisfy the specified partition value. Based on the intermediate nodes that satisfy the specified partition value, obtain the specified partition directory.

4. The metadata query method according to claim 2, characterized in that, The query results include a list of data files. Based on the time-based filtering operator, the metadata of the data files in each specified partition directory is scanned to obtain at least one matching metadata entry for the data query request, thereby obtaining the query results, including: Based on the specified time interval of the time filtering operator, the storage time information of metadata in the leaf nodes under the specified partition directory is traversed, and the metadata that meets the specified time interval is used as the matching metadata. The storage paths of each of the matching metadata are merged to merge the matching metadata and obtain at least one target data file; Based on each of the target data files, obtain the list of data files.

5. The metadata query method according to claim 4, characterized in that, After scanning the metadata of data files in each specified partition directory based on the time filtering operator to obtain at least one matching metadata of the data query request and obtain the query result, the process further includes: When the data file list includes multiple data pages, construct the association information between the data pages and obtain the associated data pages; The associated data page is sent to the data query terminal so that the data query terminal can continuously read the data page.

6. The metadata query method according to claim 1, characterized in that, The query results also include the storage characteristics of matching metadata. After obtaining the query results, the following are also included: Receive a data scanning instruction sent by a computing node, the data scanning instruction including a scanning strategy, the scanning strategy being determined based on the storage characteristics of the matching metadata; Based on the scanning strategy, the matching metadata is scanned to obtain a status report of the data block of the matching metadata.

7. A metadata query device, characterized in that, include: A receiving module is used to receive data query requests, wherein the data query requests include partition filtering operators and time filtering operators; The partition filtering module is used to scan the partition directories of the distributed file system based on the partition filtering operator to determine at least one specified partition directory. The time filtering module is used to scan the metadata of data files in each of the specified partition directories based on the time filtering operator, obtain at least one matching metadata of the data query request, and obtain query results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the metadata query method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the metadata query method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the metadata query method as described in any one of claims 1 to 6.

Images