ORC optimized data storage format and data reading and writing method based on block data

Abstract

The present invention discloses an ORC optimized data storage format and data reading and writing method based on block data. The present invention introduces Blocklet into the ORC structure to reduce the number of tasks. The query and analysis task uses Blocklet as the task input data, and reduces the number of tasks without affecting the file reading and writing performance, thereby improving the system concurrency; provides a multi-level filtering mechanism: provides Blocklet-level index and CK-level fine-grained index inside Blocklet, and realizes File-level index, Blocklet-level index, CKRouter multi-level index to improve the filtering effect. The CK-level index can be implemented using a variety of data structures, such as B+Tree or other efficient data structures such as skip lists; provides a multi-priority predicate filtering mechanism to improve the filtering effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and in particular to an ORC optimized data storage format and a data reading method based on block data. Background Art

[0002] ORC (Optimized Row Columnar) is a columnar storage format in the Hadoop ecosystem. It is derived from Apache Hive and is an Apache top-level project. It is used to reduce data storage space and speed up Hive query speed. It is widely adopted by Spark, Presto, etc. ORC is a self-describing columnar storage format. The file metadata is serialized using ProtocolBuffers. Its file format is as follows Figures 1 to 3 As shown. The existing ORC data format has the following technical problems:

[0003] 1. Data is sorted according to the order in which it is written, but not clustered according to the physical meaning. Random reads are serious during query, and the efficiency is low.

[0004] The existing implementation scheme is to store data in an orderly and aggregated manner according to the order in which the data is written, but not in an aggregated manner according to the actual physical meaning of the data and the relationship between the data, which will lead to:

[0005] (1) When reading ORC file data, the data is scattered due to disorder, resulting in more random reads and low efficiency.

[0006] (2) When reading data, the entire RowGroup data needs to be read, resulting in IO waste

[0007] 2. Low data filtering efficiency affects query performance

[0008] When reading ORC files with predicates, Stripe-level statistics are not used. Instead, the Index information of all Stripes is traversed. The RowGroup statistics saved in the Index are used to select the RowGroup to be read in each Stripe, resulting in low filtering efficiency.

[0009] 3. Lack of predicate priority management

[0010] ORC provides a predicate push-down function, but predicates have no priority, which makes it impossible to specify predicate priorities based on predicate filtering effects and give priority to using predicates with better filtering effects for calculations, resulting in low filtering efficiency.

[0011] 4. When reading data, the complete RowGroup data is returned, and the client needs to filter it again

[0012] The smallest unit of data returned by ORC is RowGroup. Even if there are only a few data items that meet the conditions, a RowGroup of data (10,000 items by default) will be returned to the client, resulting in unnecessary transmission consumption. The client needs to filter the data again, resulting in low reading efficiency.

[0013] 5. Starting query and analysis tasks by Stripe cannot support different business needs well

[0014] By default, SparkTask starts tasks in units of Stripe, but it can only start tasks at this granularity and cannot better support the task input size and task concurrency required by different business needs. Summary of the invention

[0015] In view of the problems existing in the prior art, the present invention provides an ORC optimized data storage format and a data reading and writing method based on block data. The present invention implements a block data solution based on block data modeling and ORC as the storage format to reduce storage space and improve data reading efficiency.

[0016] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0017] An ORC-optimized data storage format based on block data, using a self-describing columnar storage format, with the following data structures in order:

[0018] File Header, used to record the file format type;

[0019] Blocklet aggregates multiple data segments with the same clustering key CK feature in the ORC file into a data block;

[0020] Metadata: used to store column statistics of all Blocklets in the description file;

[0021] File Footer: used to store the Blocklet location information, Blocklet length, and Stripe information contained in the Blocklet in the description file. It is used to implement Blocklet jumps and file segmentation points for query or analysis tasks.

[0022] Postscript is used to record the file's compression algorithm, compression block size, Flie Footer length, and MetaData length.

[0023] Preferably, the clustering key CK adopts identification data with real physical meaning or data association relationship.

[0024] Preferably, the clustering key CK adopts a mobile phone number or a device number.

[0025] Preferably, the block data Blocklet is formed by continuously storing data segments Stripe that all have the same clustering key CK, and also includes a CK route at the end of each block data Blocklet.

[0026] Preferably, the CK routing indicates the data segment Stripe where each clustering key CK is located, as well as the first appearance row number and the last appearance row number in the data segment Stripe.

[0027] Preferably, the CK routing adopts a jump table or a B+Tree tree data structure.

[0028] Preferably, the data structure table of the CK routing includes Region information, device number stored in Region, file path contained in Region, and CK statistical information; wherein: the Region is an interval formed according to the value range of CK, and the Region information of each interval includes Region number, startCK, and endCK.

[0029] In addition, the present invention also provides a routing management method based on the above-mentioned ORC optimized data storage format. When querying and accessing the routing module, the Region to which it belongs and the device number stored in the Region are calculated according to the CK value to be queried, and then the device is accessed. According to the CK value and Region information, a list of files that meet the conditions is filtered out.

[0030] In addition, the present invention also provides a data writing method based on the above-mentioned ORC optimized data storage format, comprising the following steps: first, sorting the data to be written according to CK and then grouping it by Region; then, traversing to obtain each row of data, and locating the Region to which the current data belongs, and then writing by row and updating the CK route, closing the old file when the file reaches the threshold, and opening a new file to continue writing.

[0031] In addition, the present invention also provides a data reading method based on the above-mentioned ORC optimized data storage format, comprising the following steps: first, parsing the query conditions, and determining the CK predicate priority according to the predicate priority management module; then, for high-priority CK predicates, accessing the routing management module / file tail statistics / Blocklet internal CK routing in sequence according to the index granularity from coarse to fine to locate the data, and finally reading the data; for ordinary priority predicates, filtering Blocklet / Stripe / RowGroup in sequence according to the retrieval conditions to locate the data, and finally reading the data.

[0032] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0033] (1) Optimize data organization and reduce storage usage: Data is sorted by CK and aggregated into blocks, which are physically stored continuously. This can maximize the data compression effect and greatly reduce storage usage.

[0034] (2) Provide a multi-level filtering mechanism to accelerate data filtering: Provide file-level data filtering through the routing module; accelerate the filtering and positioning of internal file data through Blocklet statistics, Stripe statistics, and CK-level fine-grained indexing. By giving priority to the use of CK predicates, the data filtering speed can be accelerated.

[0035] (3) Reduce unnecessary data outbound and transmission: Through short-circuit reading of the sorting feature, there is no need to return complete RowGroup data to the client, reducing useless data transmission.

[0036] (4) Start tasks by Blocklet, reduce the number of tasks, and improve system query concurrency: FRC starts query tasks by Blocklet. One task is equivalent to multiple tasks of ORC, which reduces the number of tasks and improves system concurrency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the overall data structure of an ORC file in the prior art;

[0038] Figure 2 It is a schematic diagram of the structure of the data segment Stripe of an ORC file in the prior art;

[0039] Figure 3 It is a schematic diagram of the file tail structure of an ORC file in the prior art;

[0040] Figure 4 This is a schematic diagram of the file structure based on the ORC optimized data storage format of the present invention;

[0041] Figure 5 The data writing process based on ORC optimized data storage format described in the present invention;

[0042] Figure 6 This is the data reading process based on the ORC optimized data storage format described in the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, modifications to various equivalent forms of the present invention by those skilled in the art all fall within the scope defined by the appended claims of the present application.

[0044] like Figures 1 to 3 The figure shows the overall structure of the ORC file in the prior art. The meaning and function of each part of the file are as follows:

[0045] Ø File Header

[0046] The File Header is used to store the file type, such as "ORC" or the optimized file "FRC" of the present invention, indicating that it is an ORC file or an FRC file.

[0047] Ø Stripe

[0048] The data in the ORC file is divided into multiple segments, each segment is called a Stripe, and the data in the Stripe is stored continuously by column.

[0049] 1) Stripe Index

[0050] The statistical information and data location information of each row group after compression are stored continuously by column.

[0051] 2) Raw Data

[0052] The encoded and compressed raw data is stored continuously by column. The data of each column forms a complete stream, which is saved in a byte array at the bottom layer. The RowGroup data of each column is a fragment of the stream.

[0053] 3) Stripe Footer

[0054] Record the position and length of the complete stream data of each column.

[0055] Ø RowGroup

[0056] Indicates a certain number of data, which is the basic unit returned when ORC reads. The default value is 10000. Statistics and indexes are established for each column of data by RowGroup.

[0057] Ø Meta Data

[0058] Meta Data stores the statistical information of all columns of all Stripes in the ORC file. The statistical information of a column usually includes the min, max, sum, and count of the column.

[0059] Ø File Footer

[0060] It records the length of the data part, all column schema information, the location information of each Stripe, and file-level statistics.

[0061] Ø Postscript

[0062] The compression algorithm, compression block size, File Footer length, Meta Data length, etc. of ORC files are recorded. ORC supports common compression algorithms such as LZ4, Snappy, ZSTD, ZLIB, LZO, etc. All parts except Postscript will be compressed.

[0063] The purpose of the block data-based ORC optimized data storage format and data reading and writing method provided by the present invention is to optimize the ORC as a storage format to implement the block data solution based on block data modeling. The optimization based on the ORC structure mainly includes: (1) Introducing Blocklet in the ORC structure to reduce the number of tasks. The query and analysis tasks use Blocklet as the task input data, reducing the number of tasks without affecting the file reading and writing performance, thereby improving the system concurrency; (2) Providing a multi-level filtering mechanism: providing Blocklet-level indexes and CK-level fine-grained indexes within the Blocklet. The filtering effect is improved through File-level indexes, Blocklet-level indexes, and CKRouter multi-level indexes. Among them, the implementation of the CK-level index can adopt a variety of data structures, such as B+Tree or other efficient data structures such as skip lists; (3) Providing a multi-priority predicate filtering mechanism: supporting predicate priority, that is, supporting the setting of predicate priority and filtering operations according to the predicate priority at the same time, so as to improve the filtering effect. The following optimizations are made based on the existing ORC file, and the relevant concepts and terms are explained as follows:

[0064] 1. Terminology

[0065] Ø Block data

[0066] Block data modeling means: according to the real physical meaning of the data and the relationship between the data, the data is aggregated and stored continuously to form data blocks. For example, according to the mobile phone number, the data of the same mobile phone number is stored continuously to achieve better storage compression effect and efficient data reading efficiency.

[0067] Ø Cluster Key (CK)

[0068] The field that aggregates the data is the aggregation key. Usually we choose identification data, such as mobile phone numbers, device numbers, etc. As time goes by, these identification data will continue to accumulate data.

[0069] Ø FRC

[0070] The file format optimized and modified based on ORC in this patent is called FRC

[0071] Ø Blocklet

[0072] Multiple continuous Stripes in the same FRC file constitute a Blocklet. That is, based on the native ORC which only has Stripes, a layer of logical concept is introduced as the reading unit of the query analysis task. This can reduce the number of query analysis tasks and improve the overall query concurrency of the system.

[0073] Ø Region

[0074] Each interval is a Region, which is formed by the value of CK. Each Region contains a number, startCK, and endCK. The startCK and endCK values ​​can be the original values ​​of the CK column or the hashed values. Taking the original value as an example, if CK is a mobile phone number, we can divide the Region into the following:

[0075] region_1: [000***, 137***)

[0076] region_2: [137***, 157***)

[0077] region_3: [157***, 187***)

[0078] region_4: [187***, 999***)

[0079] 2. File Structure

[0080] like Figure 4 As shown, the present invention adds and modifies optimization items based on ORC, and the optimized file is named FRC file and recorded in File Header.

[0081] A Region contains multiple FRC files. Multiple consecutive Stripes in the same FRC file form a Blocklet. Each Blocklet stores the CK-level route, i.e., the CK Router, which indicates the Stripe where each CK is located, as well as the first and last appearance lines in the Stripe.

[0082] CK-level routing can be implemented using efficient data structures such as skip lists and B+Trees. Taking B+Trees as an example, the intermediate nodes store some CK values ​​for indexing, and the leaf nodes store all CK values ​​in the Blocklet and their corresponding location information, such as Figure 4 In the figure, the leaf node "12" means the value of CK=12, which exists in the 2nd to 12th rows of Stripe1.

[0083] In the MetaData part of the file, the column statistics of the Blocklet are saved for coarse-grained filtering of Blocklets.

[0084] The location and length of each Blocklet and the Stripes it contains are saved in the File Footer, which can be used for Blocklet jumps. The location and length of the Blocklet can be used as the file segmentation point for query or analysis tasks.

[0085] 3. Routing Management

[0086] The routing management module is used for file trimming. Before actually reading the FRC file, the routing management module is accessed first to quickly filter out useless FRC files and reduce IO waste. Routing management can be implemented in a distributed mode: each service node manages a part of the Region and the routing information of this part of the Region.

[0087] The core data structures of routing management are shown in Tables 1 and 2 below. Table 1 records the basic information of each Region and the machine on which the Region is located. Table 2 records the files contained in each Region and the CK statistics of each file.

[0088] Table 1 Global routing table

[0089] region_id start_CK end_CK RegionServer 001 00000000000 15700000000 Slave1 002 15700000000 18700000000 Slave2

[0090] Table 2 Partial routing information table on Slave1

[0091] region_id file_path ck_min ck_max 001 / hdfs / data / test1.frc 13000000100 13000000199 001 / hdfs / data / test2.frc 13000000200 13000000299

[0092] During the query process, the routing module is accessed to calculate the region to which the query belongs and the machine on which the region is located based on the CK value to be queried; then the machine is accessed to filter out the list of files that meet the conditions based on the CK value and Region information.

[0093] 4. Data writing process

[0094] like Figure 5 As shown, when writing the FRC file of the present invention, the batch of data is first sorted according to CK and grouped by Region, then each row of data is traversed to obtain the Region to which the current data belongs, and then the CKRouter is written and updated row by row. When the file reaches the threshold, the old FRC file is closed, and then the routing information is extracted from the file and reported.

[0095] 5. Data reading process

[0096] like Figure 6As shown, when the FRC file data is read in the present invention, the query condition is first parsed, and then through the predicate priority management module, if it is a high-priority predicate (CK predicate), the routing management module / file tail statistics / CK Router inside the Block are accessed in sequence according to the index granularity from coarse to fine, data is located, and finally data is read; if it is a common predicate, the native query process is followed.

[0097] Compared with the existing ORC file structure, the main optimizations of the solution of the present invention are as follows:

[0098] 1. Data aggregation and sorting

[0099] When data is written into the FRC file, it is sorted by CK key, and data with the same CK is stored continuously, which can achieve higher storage compression effect and efficient data reading. The block data modeling method means: according to the real physical meaning of the data, the association relationship of the data, and the identification field of the data (such as mobile phone number, device number), the data is aggregated and stored continuously physically. Based on the block data modeling method, we try our best to reflect the real relationship of the data to achieve higher data compression effect and better data reading speed.

[0100] 2. Improve data filtering efficiency

[0101] It provides a multi-level filtering mechanism to speed up data filtering, and implements: (1) Routing management implements file-level filtering and trimming; (2) Blocklet and Stripe statistics at the end of the FRC file implement fast Stripe filtering; (3) CK Router inside the Block implements the location of the specific CK. When querying with CK conditions, the data of the specified range of the specified Stripe can be accurately read, reducing IO interaction and useless data out of the warehouse.

[0102] 3. Reduce useless calculations and data transmission

[0103] The data is sorted by CK. When reading RowGroup data, if CK does not meet the conditions, the reading process can be interrupted to avoid reading and comparing the complete RowGroup data. The query conditions are pushed to the FRC end, and only the data that meets the conditions is returned, reducing data transmission.

[0104] 4. Provide multi-level task granularity and improve system concurrency

[0105] Stripe, based on native ORC, introduces Blocklet, which supports task division and startup based on Blocklet, so that businesses can have more task granularity options, reduce the number of tasks according to Blocklet, and improve the overall concurrency of the system.

[0106] The present invention provides a new storage format based on ORC. The core of the present invention is to use CK to make the data "aggregated into blocks and placed in order" in the file: that is, the data with the same CK is physically stored continuously and sorted by the CK key in the file.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein, but the technical essence of these modifications or replacements is still within the protection scope of the present embodiment.

Claims

1. A routing management method for ORC optimized data based on block data, wherein the ORC optimized data adopts a self-describing column storage format and has a data structure in sequence: File Header, used to record the file format type; Blocklet aggregates multiple data segments with the same clustering key CK feature in the ORC file into a data block; Metadata: used to store column statistics of all Blocklets in the description file; File Footer: used to store the Blocklet location information in the description file, the length of the Blocklet, and the Stripe information contained in the Blocklet; Postscript, used to record the file's compression algorithm, compression block size, File Footer length, and MetaData length; The clustering key CK adopts identification data with real physical meaning or data association relationship; The block data Blocklet is formed by the continuous storage of all the data segments Stripe with the same clustering key CK, and the tail of each block data Blocklet also includes a CK route; The CK route indicates the data segment Stripe where each clustering key CK is located, as well as the first and last appearing row numbers in the data segment Stripe; When querying the access routing module, the region to which the query belongs and the device number stored in the region are calculated according to the CK value to be queried, and then the device is accessed. According to the CK value and region information, a list of files that meet the conditions is filtered out.

2. According to claim 1, the ORC optimized data routing management method based on block data is characterized in that: The cluster key CK is a mobile phone number or a device number.

3. The routing management method for ORC optimized data based on block data according to claim 1 is characterized in that: The CK routing adopts a data structure of a skip list or a B+Tree tree.

4. According to claim 1, the ORC optimized data routing management method based on block data is characterized in that: The data structure table of the CK route includes Region information, device number stored in Region, file path contained in Region, and CK statistical information; wherein: the Region is an interval formed according to the value range of CK, and each interval Region information includes Region number, startCK, and endCK.

5. A method for writing data based on block data ORC optimized data, wherein the ORC optimized data adopts a self-describing column storage format and has a data structure in order: File Header, used to record the file format type; Blocklet aggregates multiple data segments with the same clustering key CK feature in the ORC file into a data block; Metadata: used to store column statistics of all Blocklets in the description file; File Footer: used to store the Blocklet location information in the description file, the length of the Blocklet, and the Stripe information contained in the Blocklet; Postscript, used to record the file's compression algorithm, compression block size, File Footer length, and MetaData length; The clustering key CK adopts identification data with real physical meaning or data association relationship; The block data Blocklet is formed by the continuous storage of all the data segments Stripe with the same clustering key CK, and the tail of each block data Blocklet also includes a CK route; The CK route indicates the data segment Stripe where each clustering key CK is located, as well as the first and last appearing row numbers in the data segment Stripe; First, the data to be written is sorted by CK and then grouped by Region. Then, each row of data is traversed to locate the Region to which the current data belongs. Then, the data is written row by row and the CK route is updated. When the file reaches the threshold, the old file is closed and a new file is opened to continue writing.

6. A method for reading ORC optimized data based on block data, wherein the ORC optimized data adopts a self-describing column storage format and has a data structure in order: File Header, used to record the file format type; Blocklet aggregates multiple data segments with the same clustering key CK feature in the ORC file into a data block; Metadata: used to store column statistics of all Blocklets in the description file; File Footer: used to store the Blocklet location information in the description file, the length of the Blocklet, and the Stripe information contained in the Blocklet; Postscript, used to record the file's compression algorithm, compression block size, File Footer length, and MetaData length; The clustering key CK adopts identification data with real physical meaning or data association relationship; The block data Blocklet is formed by the continuous storage of all the data segments Stripe with the same clustering key CK, and the tail of each block data Blocklet also includes a CK route; The CK route indicates the data segment Stripe where each clustering key CK is located, as well as the first and last appearing row numbers in the data segment Stripe; First, the query conditions are parsed and the CK predicate priority is determined according to the predicate priority management module; then, for high-priority CK predicates, the routing management module / file tail statistics / Blocklet internal CK routing are accessed in sequence according to the index granularity from coarse to fine to locate the data, and finally the data is read; for normal priority predicates, Blocklet / Stripe / RowGroup are filtered in sequence according to the search conditions to locate the data, and finally the data is read.

Images