Data processing methods, apparatus, electronic devices and systems

Abstract

This application provides a data processing method, apparatus, electronic device, and system. The data processing method includes: acquiring at least one piece of data to be processed from a distributed file system; determining the numerical range of a first field in each piece of data to be processed; dividing the numerical range into at least two sub-numerical ranges; dividing the data to be processed into at least two extended metadata files based on the at least two sub-numerical ranges; and creating new extended metadata based on the data attributes of each extended metadata file; the data attributes include the sub-numerical range in which each extended metadata file is located. This application further subdivides the data by creating new extended metadata, so that subsequent queries only need to query the data in the extended metadata file containing the field information of the first field, without needing to scan all data under the entire partition, thus improving query efficiency and performance, and avoiding the problem of high latency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to data processing methods, apparatus, electronic devices, and systems. Background Technology

[0002] HDFS (Hadoop Distributed File System) is a distributed file system that runs on general-purpose hardware and is suitable for large-scale datasets. The structured data files of the distributed file system can be mapped by Hive into partitioned tables similar to those in a database.

[0003] Currently, when querying partitioned tables, Hive suffers from poor query performance and high latency because the metadata of the partitioned tables contains a large amount of data that is not further subdivided. This necessitates scanning all data files within the entire partition. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and system to solve the problem of poor query performance or high latency caused by the lack of further data segmentation.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Obtain at least one piece of data to be processed from the distributed file system, and determine the value range of the first field of each piece of data to be processed;

[0007] Divide the numerical range into at least two sub-numerical ranges;

[0008] The data to be processed is divided into at least two extended metadata files based on at least two sub-numerical ranges;

[0009] Based on the data attributes of each extended metadata file, new extended metadata is created; the data attributes include the sub-value range of each extended metadata file, and the extended metadata is stored in a distributed file system.

[0010] In one possible implementation, at least one piece of data to be processed from the distributed file system is obtained, including:

[0011] Obtain version information for the distributed file system;

[0012] Based on version information, determine the root directory of the distributed file system;

[0013] Identify at least one partition directory in the root directory that requires data processing;

[0014] The data corresponding to each partition directory is used as the data to be processed.

[0015] In one possible implementation, the numerical range is divided into at least two sub-ranges, including:

[0016] Determine the amount of data in the root directory that needs to be processed; the amount of relevant data includes at least one of the following: the total number of partition directories and the total number of files.

[0017] Based on the amount of relevant data, a partitioning factor is determined; the partitioning factor is used to determine the range of each sub-value.

[0018] Based on the partitioning factor, the numerical range is divided into at least two sub-numerical ranges.

[0019] In one possible implementation, after creating new extended metadata based on the data attributes of each extended metadata file, it includes:

[0020] For each extended metadata file, a partition subdirectory is created; the partition subdirectory is the next level directory or the same level directory as the partition directory, and the extended metadata file is stored in the distributed file system.

[0021] Save the mapping between each extended metadata file and the partition subdirectory, and / or the mapping between the sub-value range of each extended metadata file and the partition subdirectory, to the extended metadata.

[0022] Secondly, embodiments of this application provide a data processing method, including:

[0023] Retrieve query information and extract the first field from the query information;

[0024] From the extended metadata, determine at least one extended metadata file containing the field information of the first field, and use the data from at least one extended metadata file as the data to be queried; the extended metadata is generated based on the data processing method of the first aspect, and the field information includes numerical values ​​or numerical ranges;

[0025] Based on the field information, determine the data that matches the query information from the data to be queried.

[0026] In one possible implementation, based on field information, data matching the query criteria is determined from the data to be queried, including:

[0027] Based on the field information, candidate data that matches the field information is determined from the data to be queried;

[0028] From the candidate data, select the data that matches all information in the query information except for the field information, and use this data to determine the data that matches the query information.

[0029] In one possible implementation, determining the location of the field information of the first field in at least one extended metadata file from the extended metadata includes:

[0030] From the extended metadata, determine at least one sub-value range corresponding to the field information of the first field;

[0031] Based on the correspondence between sub-value ranges and partition subdirectories in the extended metadata, determine at least one partition subdirectory corresponding to at least one sub-value range;

[0032] Determine an extended metadata file corresponding to each partition subdirectory.

[0033] Thirdly, embodiments of this application provide a data processing apparatus, including:

[0034] The acquisition module is used to acquire at least one piece of data to be processed from the distributed file system and determine the value range of the first field of each piece of data to be processed.

[0035] The first division module is used to divide the numerical range into at least two sub-numerical ranges;

[0036] The second partitioning module is used to divide the numerical range into at least two sub-numerical ranges, and based on the at least two sub-numerical ranges, divide the data to be processed into at least two extended metadata files;

[0037] A new module is created to generate new extended metadata based on the data attributes of each extended metadata file. The data attributes include the sub-value range of each extended metadata file, and the extended metadata is stored in a distributed file system.

[0038] Fourthly, embodiments of this application provide a data processing apparatus, including:

[0039] The query module is used to retrieve query information and extract the first field from the query information.

[0040] The first determining module is used to determine, from the extended metadata, at least one extended metadata file containing the field information of the first field, and to use the data of the at least one extended metadata file as the data to be queried; the extended metadata is generated based on the data processing method of the first aspect, and the field information includes numerical values ​​or numerical ranges;

[0041] The second determination module is used to determine data that matches the query information from the data to be queried based on field information.

[0042] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data processing method of the first or second aspect.

[0043] Sixthly, embodiments of this application provide a data processing system, including: a driver unit, a compilation unit, an execution unit, and a distributed file system;

[0044] The extended metadata stored within the distributed file system is generated based on the data processing method described in the first aspect;

[0045] The compilation unit, electrically connected to the driver unit, is used to receive query information sent by the driver unit, extract a first field from the query information, determine first information from extended metadata, generate first execution information based on the first information and the query information, and send it to the driver unit; the first information includes information from at least one extended metadata file containing the field information of the first field.

[0046] The execution unit, electrically connected to the driver unit, is used to receive the first execution information sent by the driver unit, and to obtain data that matches the query information from the distributed file system based on the first execution information, form the first query result, and send the first query result to the driver unit.

[0047] In one possible implementation, the data processing system also includes: a metadata database;

[0048] The compilation unit is also used to determine the second information from the metadata database when the first field is not extracted from the query information, generate the second execution information based on the second information and the query information, and send it to the driver unit; the second information includes the information of the partition directory corresponding to the query information.

[0049] The execution unit is also used to receive the second execution information sent by the driver unit, and based on the second execution information, obtain data that matches the query information from the distributed file system to form the second query result, and send the second query result to the driver unit.

[0050] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data processing method of the first or second aspect.

[0051] The beneficial effects of the technical solutions provided in this application are:

[0052] This application embodiment obtains at least one piece of data to be processed from a distributed file system, determines the numerical range of the first field of each piece of data to be processed, and divides the numerical range into at least two sub-numerical ranges. Based on these at least two sub-numerical ranges, the data to be processed can be divided into at least two extended metadata files. Thus, the data to be processed is divided into two categories based on the numerical range of the first field. Extended metadata is created based on the data attributes of each extended metadata file. Since the extended metadata is obtained based on the data attributes of each extended metadata file, during subsequent queries, at least one extended metadata file corresponding to the field information of the first field can be determined from the extended metadata. Therefore, only the data in the extended metadata file needs to be queried, instead of scanning all data under the entire partition, improving query efficiency and performance, and avoiding the problem of high latency. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0054] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0055] Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0056] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0057] Figure 4 This is a schematic diagram of the structure of a first data processing device provided in an embodiment of this application;

[0058] Figure 5 This is a schematic diagram of the structure of a second data processing device provided in an embodiment of this application;

[0059] Figure 6 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0060] Figure 7 This is a flowchart illustrating the process of converting query information into specific execution tasks, as provided in an embodiment of this application.

[0061] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0062] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0063] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” indicates implementation as “A,” or implementation as “B,” or implementation as “A and B.”

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0065] First, let's introduce and explain several terms used in this application:

[0066] SQL stands for Structured Query Language. SQL is a database query and programming language used to access, query, update, and manage relational database systems; it is also the file extension for database script files.

[0067] Hive SQL: short for HiveStructuredQuery Language, provides SQL-based query capabilities and can transform SQL into MapReduce tasks, thereby simplifying the development of distributed tasks.

[0068] ETL is an abbreviation for Extract-Transform-Load, used to describe the process of extracting, transforming, and loading data from a source to a destination. The term ETL is commonly used in data warehousing, but its application is not limited to data warehousing.

[0069] Metadata, also known as intermediary data or relay data, is data about data. It primarily describes data properties and supports functions such as indicating storage location, historical data, resource lookup, and file records. Metadata is a type of electronic catalog; to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby assisting in data retrieval.

[0070] HDFS: Hadoop Distributed File System, is characterized by high fault tolerance and is designed to be deployed on low-cost hardware. It also provides high throughput access to application data, making it suitable for applications with very large datasets.

[0071] MapReduce is a programming model for parallel computation of large-scale datasets.

[0072] Hive is a data warehouse tool based on Hadoop that can map structured data files to a database table and provide complete SQL query functionality. It can also convert SQL statements into MapReduce tasks for execution.

[0073] Research has found that with the integration of lake warehouses, building warehouses on lakes and quickly adapting to different business needs is becoming a future trend in data warehouse development. Hive, as the de facto standard for data warehouse construction in the current big data field, can map structured data files stored on the underlying distributed file system HDFS into two-dimensional tables similar to databases, and can provide a SQL-like query language (Hive SQL). This allows users familiar with SQL to perform big data queries, analysis, and ETL operations through Hive SQL, avoiding the need to write complex MapReduce tasks and improving overall user efficiency.

[0074] As the amount of data in Hive tables grows, the amount of data managed by a single table can often reach tens to hundreds of TB (Terabytes), and some can even reach the PB (PetaByte) level. In order to further improve the query efficiency of Hive SQL, such large tables are often managed in the form of partitioned tables. That is, by setting partition keys, the data is divided into different HDFS directories for storage according to the partition keys. When executing queries, the partition conditions to be queried are specified in Hive SQL, and coarse-grained partition pruning is performed, thereby avoiding scanning the entire table data and achieving the goal of speeding up the query.

[0075] However, while Hive lowered the barrier to entry for big data querying and analysis, as the amount of data managed in a single table increases, although Hive's underlying execution engine can be improved by switching from MapReduce to Tez or Spark, significant latency issues still exist in querying large partitioned tables. MapReduce, Tez, and Spark are all Hive engines.

[0076] During Hive execution, the Driver first receives the Hive SQL statement submitted by the user and hands it over to the compiler for Hive SQL syntax parsing, semantic compilation, production logic, and physical execution plan. The compiler queries the metadata database to obtain the table's metadata, generates an optimized execution plan through the optimizer, and finally executes the job using a specific execution engine, such as MapReduce, Tez, or Spark. Once the job is completed, the execution engine returns the results from the distributed file system to the driver unit, thus completing the entire query process.

[0077] In this process, the Hive metadata database stores the database name, table field names, field types, field storage types, field serialization and deserialization parsers, and table partition information for Hive tables. Specifically, the metadata SDS table stores basic HDFS storage information for Hive tables, such as the input format (INPUT_FORMAT), output format (OUTPUT_FORMAT), compression format, whether bucketing is used, and the HDFS path specified during table creation. Hive partition-related metadata is stored in the PARTITIONS, PARTITION_KEYS, and PARTITION_KEY_VALS tables. PARTITIONS stores basic information such as partition ID (Identity Document), partition name, and creation time; PARTITION_KEYS stores partition field names, field types, and field order; and PARTITION_KEY_VALS stores partition field values ​​and their order.

[0078] Due to limitations in Hive's metadata design, partition metadata information is only saved at the partition directory level. For large partition tables, the number of records corresponding to the partition metadata PARTITIONS table may reach hundreds of thousands.

[0079] During the generation and optimization of execution plans in Hive, it is necessary to frequently access data in the metadata database and traverse data files under partitions to obtain the required file-level statistics. Sometimes, the number of files under a single partition can reach tens of thousands. Scanning the partition directory files will put a lot of access pressure on the HDFS NameNode (master node), which can easily form a performance bottleneck and consume a lot of time. Ultimately, this will result in very low efficiency and high latency for Hive queries on large partition tables.

[0080] Further research revealed that in large partition table queries, Hive's metadata in its metadata database only records the corresponding partition directory, resulting in a coarse design granularity. This makes it impossible to perform fine-grained pruning of files within a partition. This coarse design leads to the need to scan all data files in the entire partition when generating and optimizing the execution plan, which is the main reason for its poor query performance and high latency in large partition table queries. This is the main drawback of the current technology.

[0081] The data processing methods, apparatus, electronic devices, and systems provided in this application are intended to solve the above-mentioned technical problems of the prior art.

[0082] This application provides a data processing method, see [link to relevant documentation] Figure 1 As shown, it includes steps S101 to S104.

[0083] S101. Obtain at least one piece of data to be processed from the distributed file system, and determine the value range of the first field of each piece of data to be processed.

[0084] Optionally, the first field is a numeric field, such as: order amount.

[0085] In some embodiments, step S101, obtaining at least one piece of data to be processed from the distributed file system, includes:

[0086] Get the version information of the distributed file system.

[0087] Based on version information, determine the root directory of the distributed file system.

[0088] Identify at least one partition directory in the root directory that requires data processing.

[0089] The data corresponding to each partition directory is used as the data to be processed.

[0090] Optionally, obtaining the version information of the distributed file system means obtaining the current version information to obtain the current root directory.

[0091] Optionally, the root directory includes multiple partition directories. Determining at least one partition directory within the root directory that requires data processing may include:

[0092] Determine whether at least one partition directory meets the set partition conditions. If it meets the set partition conditions, then determine whether the partition directory needs to be processed.

[0093] Optionally, the partitioning conditions set include at least one of the following: the data file corresponding to the partition directory has partitionable information; or the partition directory can generate a subdirectory of the next level partition. Partitioning conditions can also be set according to actual application, implemented by the user by identifying the partition directory and the data file corresponding to the partition directory.

[0094] Optionally, each partition directory corresponds to at least one data file, and the data corresponding to each partition directory is all the data in the data file corresponding to the partition directory.

[0095] Specifically, determining the numerical range of the first field of each piece of data to be processed can include:

[0096] Determine the minimum and maximum values ​​of the first field of each piece of data to be processed;

[0097] The range of values ​​for the first field is determined based on the minimum and maximum values.

[0098] Optionally, the minimum and maximum values ​​can be used as the two endpoints of the numerical range. For example, the numerical range of the first field is 0-1000.

[0099] S302. Divide the numerical range into at least two sub-numerical ranges.

[0100] In some possible implementations, dividing the numerical range into at least two sub-ranges may include:

[0101] Based on the range of values, determine at least one intermediate value;

[0102] Divide the numerical range into at least two sub-ranges based on the intermediate value.

[0103] Optionally, for example, if the numerical range is 0-1000, and the intermediate values ​​are determined to be 200, 400, 600, and 800, then the numerical range can be divided into five sub-numerical ranges: 0-200, 201-400, 401-600, 601-800, and 801-1000. Based on the field value of the first field of each data point to be processed, the data is divided into five categories, forming five extended metadata files. All sub-numerical ranges are arranged sequentially. Excluding the last sub-numerical range, the intermediate value of each sub-numerical range is the maximum value. Excluding the first and last sub-numerical ranges, the minimum value of each sub-numerical range is the intermediate value plus a preset value, which can be either 1 or 0.

[0104] Optionally, in step S302, the numerical range is divided into at least two sub-numerical ranges, including:

[0105] Determine the amount of data in the root directory that needs to be processed; the amount of data includes at least one of the following: the total number of partition directories and the total number of files.

[0106] Based on the amount of relevant data, a partitioning factor is determined; the partitioning factor is used to determine the range of each sub-value.

[0107] Based on the partitioning factor, the numerical range is divided into at least two sub-numerical ranges.

[0108] Optionally, the total number of files is the same as the total number of data files, with each partition directory corresponding to at least one data file.

[0109] Optionally, the partitioning factor is used to determine the number of sub-ranges, and each sub-range is determined based on the range of values ​​and the number of sub-ranges.

[0110] Optionally, based on a partitioning factor, dividing the numerical range into at least two sub-ranges may include:

[0111] The number of sub-value ranges is determined based on the partitioning factor;

[0112] Based on the number of determined sub-range values, at least one intermediate value is determined;

[0113] Divide the numerical range into at least two sub-ranges based on the intermediate value.

[0114] Optionally, based on the number N of the determined sub-ranges, N-1 intermediate values ​​are selected from the minimum and maximum values ​​of the ranges to form N sub-ranges. The intermediate values ​​serve as endpoints of the sub-ranges. When selecting intermediate values, efforts are made to ensure that the intervals of each sub-range are the same or similar, as in the example above where the range is 0-1000. This facilitates the reasonable division of the data to be processed, further improving query efficiency.

[0115] Optionally, after determining the root directory of the distributed file system based on version information, the process may also include: determining the total number of fields. The total number of fields is used as identification information and may not participate in the data processing.

[0116] S303. Based on at least two sub-numerical ranges, divide the data to be processed into at least two extended metadata files.

[0117] Optionally, in step S303, dividing the data to be processed into at least two extended metadata files based on at least two sub-numerical ranges may include:

[0118] Determine the sub-value range of the first field of each data point to be processed. Based on the sub-value range of the first field of each data point, divide the data to be processed into at least two categories of data, and store each category of data in an extended metadata file.

[0119] Optionally, the metadata file can be extended to a JSON file. JSON is an open standard data interchange format based on a subset of JavaScript syntax.

[0120] Optionally, each extended metadata file is a collection of a type of data, and the data within the extended metadata file can be stored in categories as multiple data files.

[0121] S304. Based on the data attributes of each extended metadata file, create new extended metadata; the data attributes include the sub-value range where each extended metadata file is located, and the extended metadata is stored in a distributed file system.

[0122] Optionally, the extended metadata is generated based on the data attributes of each extended metadata file. The extended metadata can be an extended metadata table, which includes the correspondence between each extended metadata file and the sub-numerical range.

[0123] Optionally, the correspondence information between each extended metadata file and sub-numerical range can be in the form of the filename of the extended metadata file and the correspondence between the sub-numerical range.

[0124] The embodiments of this application can store extended metadata based on a distributed file system, which can effectively reduce the query performance loss caused by centralized storage of metadata, thereby achieving the effect of optimizing queries.

[0125] In some embodiments, after creating new extended metadata based on the data attributes of each extended metadata file in step S304, the process includes:

[0126] For each extended metadata file, a partition subdirectory is created; the partition subdirectory is the next level directory or the same level directory as the partition directory, and the extended metadata file is stored in the distributed file system.

[0127] Save the mapping between each extended metadata file and the partition subdirectory, and / or the mapping between the sub-value range of each extended metadata file and the partition subdirectory, to the extended metadata.

[0128] Optionally, the extended metadata stores the correspondence between the extended metadata file and the partition subdirectory and / or the correspondence between the sub-value range and the partition subdirectory. This facilitates subsequent queries by determining at least one partition subdirectory where the field information is located based on the field information of the first field, finding the extended metadata file corresponding to the partition subdirectory in the distributed file system, and then scanning the data in the extended metadata file to obtain data that matches the query information.

[0129] Optionally, the extended metadata includes the number of extended metadata files and statistical descriptions of the columns stored in each extended metadata file, such as the maximum value, minimum value, and type of the column.

[0130] Optionally, the partition directory can be a time-based directory, for example, partitioned by year, with partition directories for 2020, 2021, and 2022 respectively. Each year's partition directory corresponds to at least one data file. Within each year's partition directory, further partitioning is performed based on the numerical range of the first field, resulting in at least two partition subdirectories and corresponding extended data files.

[0131] Optionally, in this embodiment of the application, when loading Parquet or ORC type data files into a Hive partitioned table, the column statistics information of the data file, such as column type, encoding / decoding type, null values, maximum values, minimum values, etc., are uniformly stored in an extended metadata file. Parquet type data files are a compressed and efficient columnar data representation format that can be used in any project within the Hadoop Distributed File System; it is a data storage format. ORC (Optimized RowColumnar) type data files can be applied in the Hadoop Distributed File System and are also an efficient data storage format.

[0132] Optionally, based on the data attributes of each extended metadata file, after creating new extended metadata, it can include:

[0133] In response to an update operation on the extended metadata file or a partition subdirectory, update the relevant information in the extended metadata accordingly. The update operation includes at least one of the following: adding, deleting, or modifying. The relevant information includes at least one of the following: the correspondence between the extended metadata file and the sub-value range, the correspondence between the extended metadata file and the partition subdirectory, and the correspondence between the sub-value range and the partition subdirectory. Updating the relevant information in the extended metadata refers to updating the relevant information corresponding to the addition, deletion, or modification.

[0134] Optionally, during the data loading process, Hive obtains the sub-value range corresponding to the partition directory where the corresponding data file is located through the columnar file API (Application Program Interface), and updates the data in the data file to the extended metadata file in a unified manner.

[0135] Optionally, Hive in Figure 2 Step 3, the logical execution plan generation stage, shows how to obtain extended metadata such as the upper and lower limits of the field value and the field encoding type in each data file under the partition directory corresponding to the field involved in this query. Figure 2 In the fourth step of the logical execution plan, based on table scan operations such as Select and Filter operation nodes, the field information of the corresponding first field in the extended metadata is used to obtain the partition subdirectories involved in the field value range, as well as the extended metadata files under the corresponding partition subdirectories. This allows for fine-grained pruning of partition files, avoiding the operation of traversing all partition files, and effectively reducing the number of partition directory files scanned for large partition tables.

[0136] Optionally, see Figure 2 As shown in the figure, this application embodiment provides a data processing method, including steps S201 to S209.

[0137] S201. Obtain the version information of the distributed file system.

[0138] Optionally, obtaining the version information of the distributed file system means obtaining the current version information to obtain the current root directory.

[0139] S202. Based on the version information, determine the root directory of the distributed file system, and then execute steps S203 and S205.

[0140] Optionally, the root directory may include multiple partition directories.

[0141] S203. Identify at least one partition directory in the root directory that requires data processing.

[0142] Identify at least one partition directory in the root directory that requires data processing, including:

[0143] Determine whether at least one partition directory meets the set partition conditions. If it meets the set partition conditions, then determine whether the partition directory needs to be processed.

[0144] Optionally, the partitioning conditions set include at least one of the following: the data file corresponding to the partition directory has partitionable information; or the partition directory can generate a subdirectory of the next level partition. Partitioning conditions can also be set according to the actual application, implemented by the user by identifying the partition directory and the data file corresponding to the partition directory.

[0145] Optionally, each partition directory corresponds to at least one data file, and the data corresponding to each partition directory is all the data in the data file corresponding to the partition directory.

[0146] S204. Take the data corresponding to each partition directory as the data to be processed, and then execute step S208.

[0147] S205. Determine the amount of data in the root directory that needs to be processed; the amount of data includes at least one of the following: the total number of partition directories and the total number of files.

[0148] Optionally, the total number of files is the same as the total number of data files, with each partition directory corresponding to at least one data file.

[0149] S206. Based on the amount of relevant data, determine the partitioning factor; the partitioning factor is used to determine the range of each sub-value.

[0150] Optionally, the partitioning factor is used to determine the number of sub-ranges, and each sub-range is determined based on the range of values ​​and the number of sub-ranges.

[0151] S207. Based on the partitioning factor, divide the numerical range into at least two sub-numerical ranges.

[0152] S208. Based on at least two sub-numerical ranges, divide the data to be processed into at least two extended metadata files.

[0153] Optionally, step S208 is the same as step S103, and will not be described again here.

[0154] S209. Based on the data attributes of each extended metadata file, create new extended metadata; the data attributes include the sub-value range where each extended metadata file is located, and the extended metadata is stored in a distributed file system.

[0155] Optionally, step S209 is the same as step S104, and will not be described again here.

[0156] The data extraction method in this application provides a novel approach. When Hive loads partitioned table data, extended metadata is created at the partition level. To avoid performance issues caused by centralized metadata storage, the extended metadata is uniformly stored in a distributed file system, and the extended metadata file itself is also saved there. During Hive compilation and execution, this extended metadata can be used for efficient file filtering and operator pushdown, avoiding extensive scanning of data files in numerous partition directories, thereby improving query efficiency.

[0157] This application provides a data processing method, see [link to relevant documentation] Figure 3 As shown, the data processing method includes steps S301 to S303.

[0158] S301. Obtain query information and extract the first field from the query information.

[0159] Optionally, the query information can be a Hive SQL statement, with the first field being a numeric type.

[0160] Optionally, extracting the first field from the query information includes: extracting the first field and the field information of the first field from the query information.

[0161] Optionally, the first field is extracted from the query information, which means that the query information includes the first field and can be queried from the extended metadata based on the field information of the first field.

[0162] S302. From the extended metadata, determine at least one extended metadata file containing the field information of the first field, and use the data of the at least one extended metadata file as the data to be queried; the extended metadata is generated based on the data processing method of any of the embodiments of this application described above, and the field information includes numerical values ​​or numerical ranges.

[0163] In some embodiments, determining from the extended metadata at least one extended metadata file where the field information of the first field resides includes:

[0164] From the extended metadata, determine at least one sub-value range corresponding to the field information of the first field;

[0165] Based on the correspondence between sub-value ranges and partition subdirectories in the extended metadata, determine at least one partition subdirectory corresponding to at least one sub-value range;

[0166] Determine an extended metadata file corresponding to each partition subdirectory.

[0167] Optionally, the extended metadata stores the correspondence between extended metadata files and partition subdirectories, and / or the correspondence between sub-value ranges and partition subdirectories. Based on the field information of the first field, at least one partition subdirectory containing the field information is determined, and then an extended metadata file corresponding to each partition subdirectory is determined, thereby obtaining the data to be queried.

[0168] Optionally, each extended metadata file includes multiple data sets, and all the data from at least one extended metadata file constitutes the data to be queried.

[0169] Optionally, the extended metadata in this application embodiment stores the correspondence between extended metadata files and partition subdirectories and / or the correspondence between sub-value ranges and partition subdirectories. By sequentially determining the sub-value ranges and partition subdirectories, at least one extended metadata file containing the field information can be determined, thereby determining the data to be queried.

[0170] S303. Based on field information, determine the data that matches the query information from the data to be queried.

[0171] In some embodiments, based on field information, determining data that matches the query information from the data to be queried includes:

[0172] Based on the field information, candidate data that matches the field information is determined from the data to be queried;

[0173] From the candidate data, select the data that matches all information in the query information except for the field information, and use this data to determine the data that matches the query information.

[0174] Based on the technical solution of this application's embodiments, the data processing process of this application is further illustrated below through an example. There is a petabyte-level order table with five fields: Order ID (string type), User ID (string type), Product ID (string type), Order Amount (numeric type), and Order Time (date type). The table is partitioned by day, and the corresponding partition directories and data files are divided by date. Because the table data is very large, there are many data files under a single partition, such as 100,000 Parquet type files. Extended metadata needs to be generated after the data is loaded.

[0175] Optionally, if the order amount is a numeric type, it is used as the first field, and an extended-info node is set in the encoding. For non-numeric extended metadata fields, there is no corresponding extended-info node. For numeric fields such as order amount, there is an extended node that identifies the extended metadata file name within the corresponding numeric value range.

[0176] For example, in the extended metadata, the sub-value ranges are 1-1000, 1001-2000, and 2001-5000, and each sub-value range corresponds to an extended metadata file and a partition subdirectory.

[0177] For the query "March 11, 2022, total value greater than 1200 and less than 2000", the Hive compiler can quickly locate the partition subdirectories and extended metadata files based on this filtering condition and the information of the extended nodes during optimization. Within the date-based partition directories or subdirectories, it can then identify data that matches the query information. For example, based on the above query, each extended metadata file can be identified. These extended metadata files include data012.parquet, data033.parquet, and data018.parquet. Only these extended metadata files need to be scanned to identify the data that meets the query information, thus avoiding the need to scan all 100,000 files in the partition to achieve optimization.

[0178] This application's embodiment addresses the current landscape of big data warehouses primarily based on Hive. By further partitioning data according to numerical ranges and creating extended metadata, it effectively improves the query response speed of large partitioned Hive tables. During the optimization phase, it provides more accurate decision-making basis for the Hive execution plan, thereby effectively hitting data files in the partition directory and row group areas within those files during the physical execution phase. This avoids numerous invalid file scans and overall reads, effectively solving the query efficiency problem of large partitioned tables. This application's embodiment is applicable to performance improvement and optimization of large table queries for integrated lakeware warehouses.

[0179] This application provides a first data processing apparatus, see [link to previous document]. Figure 4 As shown, the first data processing device 40 includes: an acquisition module 401, a first division module 402, a second division module 403, and a new creation module 404.

[0180] The acquisition module 401 is used to acquire at least one piece of data to be processed from the distributed file system and determine the value range of the first field of each piece of data to be processed.

[0181] The first partitioning module 402 is used to divide the numerical range into at least two sub-numerical ranges.

[0182] The second partitioning module 403 is used to partition the data to be processed into at least two extended metadata files based on at least two sub-numerical ranges.

[0183] The new module 404 is used to create new extended metadata based on the data attributes of each extended metadata file; the data attributes include the sub-value range of each extended metadata file, and the extended metadata is stored in a distributed file system.

[0184] Optionally, the acquisition module 401 is used to acquire version information of the distributed file system, determine the root directory of the distributed file system based on the version information, determine at least one partition directory in the root directory that needs to be processed, and take the data corresponding to each partition directory as the data to be processed.

[0185] Optionally, the first partitioning module 402 is used to determine the amount of relevant data in the root directory that needs to be processed; the amount of relevant data includes at least one of the following: the total number of partition directories and the total number of files. Based on the amount of relevant data, a partitioning factor is determined; the partitioning factor is used to determine each sub-numerical range, and based on the partitioning factor, the numerical range is divided into at least two sub-numerical ranges.

[0186] Optionally, the new module 404 is used to create a partition subdirectory based on each extended metadata file; the partition subdirectory is the next level directory or the same level directory of the partition directory, and the extended metadata file is stored in the distributed file system; the correspondence between each extended metadata file and the partition subdirectory, and / or the correspondence between the sub-numerical range of each extended metadata file and the partition subdirectory is saved to the extended metadata.

[0187] The first data processing device in this embodiment can execute any of the data processing methods for creating new extended metadata provided in the embodiments of this application. Their implementation principles are similar and will not be described in detail here.

[0188] This application provides a second data processing apparatus, see [link to previous document]. Figure 5 As shown, the second data processing device 50 includes: a query module 501, a first determination module 502, and a second determination module 503.

[0189] The query module 501 is used to obtain query information and extract the first field from the query information.

[0190] The first determining module 502 is used to determine, from the extended metadata, at least one extended metadata file containing the field information of the first field, and to use the data of the at least one extended metadata file as the data to be queried; the extended metadata is generated based on the data processing method of the first aspect, and the field information includes numerical values ​​or numerical ranges.

[0191] The second determining module 503 is used to determine data that matches the query information from the data to be queried based on field information.

[0192] Optionally, the first determining module 502 is used to determine at least one sub-value range corresponding to the field information of the first field from the extended metadata; determine at least one partition subdirectory corresponding to the at least one sub-value range based on the correspondence between the sub-value range in the extended metadata and the partition subdirectory; and determine an extended metadata file corresponding to each partition subdirectory.

[0193] Optionally, the second determining module 503 is used to determine candidate data that matches the field information from the data to be queried based on the field information; and to determine data that matches other information in the query information besides the field information from the candidate data, as the data that matches the query information.

[0194] The second data processing device in this embodiment can execute any of the data processing methods for data query provided in the embodiments of this application. Their implementation principles are similar and will not be described in detail here.

[0195] The data processing method of this application will be further described below from the perspective of system architecture. An embodiment of this application provides a data processing system, see below. Figure 6 As shown, the data processing system includes: a driver unit, a compiler unit, an execution unit (including MapReduce, Tez, and Spark), and a distributed file system.

[0196] The extended metadata stored within the distributed file system is generated based on the data processing method of any embodiment of this application. The specific method for generating the extended metadata will be described in further detail in the subsequent data processing methods.

[0197] The compilation unit is electrically connected to the driver unit. The compilation unit is used to receive query information sent by the driver unit, extract the first field from the query information, determine the first information from the extended metadata, generate the first execution information based on the first information and the query information, and send it to the driver unit. The first information includes information from at least one extended metadata file where the field information of the first field is located.

[0198] The execution unit is electrically connected to the driver unit. The execution unit is used to receive the first execution information sent by the driver unit, and obtain data that matches the query information from the distributed file system based on the first execution information to form the first query result, and send the first query result to the driver unit.

[0199] Optionally, the drive unit may be a driver used to receive query information.

[0200] The query information can be a Hive SQL statement submitted by the user.

[0201] Optionally, the compilation unit may be a compiler used to perform syntax parsing, semantic compilation, production logic, and physical execution planning on Hive SQL statements.

[0202] Optionally, the execution unit uses the Hive engine to execute the job. The Hive engine includes: the default MapReduce (i.e., MR shown in the figure), Tez, or Spark. If the engine is not changed, the default Hive engine is MR.

[0203] Optionally, the extended metadata file data is stored in a distributed file system.

[0204] Optionally, the information of at least one extended metadata file containing the field information of the first field can be obtained by: determining at least one sub-value range containing the field information of the first field, and determining at least one extended metadata file containing the field information of the first field based on the correspondence between the sub-value range in the extended metadata and the extended metadata file, as the first information.

[0205] Optionally, the extended metadata is generated based on the data attributes of each extended metadata file. The extended metadata includes the correspondence between the extended metadata file and the sub-numerical range, that is, the sub-numerical range in which each extended metadata file is located.

[0206] Optionally, extended metadata also belongs to metadata, which refers to an extension of the original metadata. Extended metadata can be an extended metadata table, and the second information is data obtained from the extended metadata table.

[0207] In some embodiments, see Figure 6 As shown, the data processing system also includes: MetaStore.

[0208] The compilation unit is also used to determine the second information from the metadata database when the first field is not extracted from the query information, generate the second execution information based on the second information and the query information, and send it to the driver unit; the second information includes the information of the partition directory corresponding to the query information;

[0209] The execution unit is also used to receive the second execution information sent by the driver unit, and based on the second execution information, obtain data that matches the query information from the distributed file system to form the second query result, and send the second query result to the driver unit.

[0210] Optionally, the metadata database stores information about the partition directories of the distributed file system of the Hive table. That is, what is obtained from the metadata database includes the metadata of the partition directory where the query information is located, as secondary information.

[0211] Optionally, the first field is a numeric field, such as: order amount.

[0212] Optionally, the metadata database is used to store metadata, which can be a metadata table, and the second information is data obtained from the metadata table.

[0213] This application embodiment can determine the first information from extended metadata when the query information includes a first field, thereby identifying the information of at least one extended metadata file containing the field information of the first field. This allows the execution unit to scan and query the data within the extended metadata files in the distributed file system based on the identified at least one extended metadata file, thus determining the data that matches the query information and forming a first query result. This application embodiment improves query efficiency by creating new extended metadata in the distributed file system, enabling faster querying of query information including the first field.

[0214] In this embodiment of the application, when the query information does not include the first field, the partition directory information corresponding to the query information is determined in the original metadata database. Thus, all data under the partition corresponding to the partition directory information can be found in the distributed file system. Therefore, when the query information does not include the first field, the second query result of data matching the query information can also be obtained from the distributed file system through the metadata database query.

[0215] Optionally, the query information can be a Hive SQL statement. The process of converting Hive SQL into a specific execution task involves stages and the output of each stage.

[0216] Optionally, for the process of converting query information into specific execution tasks, see [link to relevant documentation]. Figure 7As shown, a Hive SQL statement first needs to undergo syntactic and lexical analysis to generate an Abstract Syntax Tree (AST); the second step, semantic analysis, generates a Query Block; the third step, generating a logical execution plan, involves traversing the Query Block and executing the operation tree, which includes at least one of the following: TableScanOperator, SelectOperator, FilterOperator, JoinOperator, GroupByOperator, or ReduceSinkOperator; the fourth step, optimizing the logical execution plan's OP tree, involves projection pruning, predicate pushdown, merging connected Select and Filter operations into a single operation tree node, and reducing unnecessary operations; the fifth step, generating a physical execution plan, transforms the optimized logical plan's operation tree into tasks corresponding to specific execution engines; and the sixth step, after optimizing the physical execution plan (Task tree), such as removing unnecessary ReduceSinkOperators and reducing the number of user-submitted jobs, generates an optimized physical plan (Task tree) and submits it for execution.

[0217] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the data processing method of any embodiment of this application.

[0218] In one alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0219] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0220] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0221] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0222] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0223] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data processing method of any embodiment of this application.

[0224] It should be noted that the computer-readable medium of this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0225] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0226] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the version information of the distributed file system; Based on the version information, the root directory of the distributed file system is determined; Identify at least one partition directory in the root directory that requires data processing; use the data corresponding to each partition directory as the data to be processed; Determine the minimum and maximum values ​​of the first field of each piece of data to be processed, and use the minimum and maximum values ​​as the two endpoints of the numerical range of the first field of the data to be processed; the first field is a numeric field. Divide the numerical range into at least two sub-numerical ranges; Determine the sub-value range of the first field of each data in the data to be processed, and based on the sub-value range of the first field of each data, divide the data to be processed into at least two types of data, with each type of data stored in an extended metadata file; Based on the data attributes of each of the extended metadata files, new extended metadata is created; the data attributes include the sub-value range where each of the extended metadata files is located, and the extended metadata is stored in the distributed file system; Based on each of the aforementioned extended metadata files, a partition subdirectory is created; The partition subdirectory is the next level directory or the same level directory of the partition directory, and the extended metadata file is stored in the distributed file system; the correspondence between each extended metadata file and the partition subdirectory, and / or the correspondence between the sub-numerical range of each extended metadata file and the partition subdirectory, are saved to the extended metadata.

2. The data processing method according to claim 1, characterized in that, The step of dividing the numerical range into at least two sub-numerical ranges includes: Determine the amount of relevant data in the root directory that needs to be processed; the amount of relevant data includes at least one of the following: the total number of partition directories and the total number of files; Based on the amount of relevant data, a partitioning factor is determined; the partitioning factor is used to determine the range of each of the sub-values. Based on the partitioning factor, the numerical range is divided into at least two sub-numerical ranges.

3. A data processing method, characterized in that, include: Obtain query information and extract the first field from the query information; From the extended metadata, determine at least one extended metadata file containing the field information of the first field, and use the data from the at least one extended metadata file as the data to be queried; The extended metadata is generated based on the data processing method as described in any one of claims 1-2, and the field information includes numerical values ​​or numerical ranges; Based on the field information, data that matches the query information is determined from the data to be queried.

4. The data processing method according to claim 3, characterized in that, The step of determining data matching the query information from the data to be queried based on the field information includes: Based on the field information, candidate data that matches the field information is determined from the data to be queried; From the candidate data, data that matches other information in the query information besides the field information is selected as the data that matches the query information.

5. The data processing method according to claim 3, characterized in that, Determining from the extended metadata the location of at least one extended metadata file containing the field information of the first field includes: From the extended metadata, determine at least one sub-value range corresponding to the field information of the first field; Based on the correspondence between the sub-value ranges and partition subdirectories in the extended metadata, determine at least one partition subdirectory corresponding to the at least one sub-value range; Determine an extended metadata file corresponding to each of the aforementioned partition subdirectories.

6. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the version information of the distributed file system; Based on the version information, the root directory of the distributed file system is determined; Identify at least one partition directory in the root directory that requires data processing; use the data corresponding to each partition directory as the data to be processed; Determine the minimum and maximum values ​​of the first field of each piece of data to be processed, and use the minimum and maximum values ​​as the two endpoints of the numerical range of the first field of the data to be processed; the first field is a numeric field. The first division module is used to divide the numerical range into at least two sub-numerical ranges; The second partitioning module is used to determine the sub-value range of the first field of each data in the data to be processed, and to partition the data to be processed into at least two types of data based on the sub-value range of the first field of each data, with each type of data stored in an extended metadata file; A new module is created to generate new extended metadata based on the data attributes of each extended metadata file; the data attributes include the sub-value range of each extended metadata file, and the extended metadata is stored in the distributed file system; Create a partition subdirectory based on each extended metadata file; A partition subdirectory is a subdirectory or a sibling directory of the partition directory, and the extended metadata file is stored in a distributed file system. Save the mapping between each extended metadata file and the partition subdirectory, and / or the mapping between the sub-value range of each extended metadata file and the partition subdirectory, to the extended metadata.

7. A data processing apparatus, characterized in that, include: The query module is used to obtain query information and extract the first field from the query information; The first determining module is used to determine, from the extended metadata, at least one extended metadata file containing the field information of the first field, and to use the data of the at least one extended metadata file as the data to be queried; The extended metadata is generated based on the data processing method as described in any one of claims 1-2, and the field information includes numerical values ​​or numerical ranges; The second determining module is used to determine data that matches the query information from the data to be queried based on the field information.

8. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of the data processing method according to any one of claims 1-2 or 3-5.

9. A data processing system, characterized in that, include: Driver unit, compilation unit, execution unit, and distributed file system; The extended metadata stored in the distributed file system is generated based on the data processing method as described in any one of claims 1-2; The compilation unit, electrically connected to the driver unit, is used to receive query information sent by the driver unit, extract a first field from the query information, determine first information from the extended metadata, generate first execution information based on the first information and the query information, and send it to the driver unit; the first information includes information from at least one extended metadata file containing the field information of the first field. The execution unit is electrically connected to the driving unit and is used to receive the first execution information sent by the driving unit, obtain data that matches the query information from the distributed file system based on the first execution information, form a first query result, and send the first query result to the driving unit.

10. The data processing system according to claim 9, characterized in that, Also includes: Metadatabase; The compilation unit is further configured to, when the first field is not extracted from the query information, determine the second information from the metadata database, generate the second execution information based on the second information and the query information, and send it to the driver unit; the second information includes information about the partition directory corresponding to the query information; The execution unit is further configured to receive second execution information sent by the driver unit, and obtain data matching the query information from the distributed file system based on the second execution information to form a second query result, and send the second query result to the driver unit.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method according to any one of claims 1-2 or 3-5.

Images