Hardware-implemented file reader

Abstract

A hardware-implemented file reader (20) includes an interface (32), a plurality of hardware-implemented column readers (36), and a record reconstructor (40). The interface is configured to access a file (24) comprising a plurality of records. Records store values ​​according to a nested structure supporting optional and repeating values. The file is stored in a column format with multiple columns, each column storing (i) a compressed value and (ii) corresponding compressed structure information that associates the values ​​in the column with the nested structure of the records. Each column reader is configured to be assigned to a corresponding selected column and to read and decompress the values ​​and structure information from at least a portion of the selected column. The record reconstructor is configured to reconstruct one or more records from the records based on at least a portion of the columns read by the column readers and output the reconstructed records.

Description

Invention Field

[0001] This invention relates generally to digital hardware and microprocessor design, and more particularly to methods and systems for hardware-based file reading. Background of the Invention

[0003] Apache Parquet is a popular columnar data storage format. Parquet files have a complex hierarchical structure and support various compression and encoding modes. Several attempts have been made to process Parquet files in hardware. For example, Van Leeuwen et al. described a Parquet to Arrow converter in their paper "High-throughput conversion of Apache Parquet files to Apache Arrow in-memory format using FPGAs" at CompSys 2019 on June 4, 2019. Invention Overview

[0005] The embodiments of the present invention described herein provide a hardware-implemented file reader comprising an interface, a plurality of hardware-implemented column readers, and a hardware-implemented record reconstructor. The interface is configured to access a file comprising a plurality of records. Records store values ​​according to a nested structure supporting optional and repeating values. The file is stored in a column format with multiple columns, each column storing (i) a compressed value and (ii) corresponding compressed structure information associating the values ​​in the column with the nested structure of the records. Each column reader is configured to be assigned to a corresponding selected column, and to read and decompress the values ​​and structure information from at least a portion of the selected column. The record reconstructor is configured to reconstruct one or more records based on at least a portion of the column read by the column reader, and to output the reconstructed records.

[0006] In some embodiments, the column format is Parquet format, and the structural information includes repetition levels and definition levels of the values. In one embodiment, the record reconstructor is configured to apply backpressure to one or more column readers to align the corresponding outputs of the column readers to belong to no more than a predefined number of adjacent records.

[0007] In an example embodiment, the record reconstructor is configured to determine the appropriate data size that each record needs to obtain from each column reader, and to maintain alignment between column readers by obtaining the determined data size from each column reader. In the disclosed embodiment, a given column reader is configured to align at least some of the decompressed values ​​with the corresponding decompressed structural information before reading and decompressing subsequent values ​​and subsequent structural information from the selected columns.

[0008] In some embodiments, a given column reader includes a value reader configured to read and decompress values ​​of a selected column and one or more structure information readers configured to read and decompress structural information of the selected column. In an example embodiment, the structure information includes a repetition level and a definition level of the values, and the structure information reader includes a repetition level reader configured to read and decompress the repetition level and a definition level reader configured to read and decompress the definition level. In another embodiment, a given column reader includes a single reader configured to alternately read and decompress both the values ​​of the selected column and the structural information of the selected column.

[0009] In yet another embodiment, when reconstructing nested records, the record reconstructor is configured to decompose one or more nesting levels of the nested record, thereby outputting multiple records in place of the nested record. In yet another embodiment, in response to a request to reconstruct a set of columns greater than the number of column readers, the record reconstructor is configured to reconstruct and output two or more partial record sets, each partial record set corresponding to a corresponding subset of the requested set of columns.

[0010] In some embodiments, the file reader further includes a hardware-implemented dictionary circuit configured to read a dictionary from the file, where values ​​are represented by corresponding keys, and subsequently translate the keys read from the file into corresponding values ​​so that the translated values ​​are placed in a reconstructed record. In one embodiment, the file reader includes a dictionary data structure that maps keys to corresponding values, and the dictionary circuit is configured to populate the dictionary data structure as the dictionary is read from the file. In one embodiment, the dictionary circuit is configured to store a portion of the dictionary in a cache.

[0011] In some embodiments, one or more column readers and record reconstructors are configured to modify one or more values ​​read from a file. In example embodiments, the record reconstructor is configured to output only the modified value, or both the value read from the file and the modified value. In one embodiment, the record reconstructor is configured to specify the modification of the value based on a received query.

[0012] In another embodiment, the record reconstructor is configured to filter records based on one or both of the following: (i) one or more defined criteria for the values, and (ii) a received query. In the disclosed embodiments, the record reconstructor includes multiple processing engines configured to reconstruct multiple corresponding records simultaneously.

[0013] In some embodiments, the record reconstructor is configured to reconstruct multiple record streams in parallel. In one embodiment, the record reconstructor is configured to reconstruct multiple record streams independently of each other. In the disclosed embodiments, the record reconstructor is configured to apply reverse pressure to the column reader only for a selected subset of the record streams.

[0014] According to embodiments of the present invention, a method for hardware-implemented file reading is also provided. The method includes accessing a file using a plurality of hardware-implemented column readers. The file includes a plurality of records, and these records store values ​​according to a nested structure supporting optional and repeating values. The file is stored in a column format with multiple columns, each column storing (i) compressed values ​​and (ii) corresponding compressed structure information associating the values ​​in the column with the nested structure of the records. Each column reader is assigned to a corresponding selected column and reads and decompresses both the values ​​and the structure information from at least a portion of the selected column. A hardware-implemented record reconstructor reconstructs one or more of the records based on at least a portion of the columns read by the column readers and outputs the reconstructed records.

[0015] The invention will be more fully understood from the following detailed description of embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a block diagram schematically illustrating a hardware implementation of a Parquet reader according to an embodiment of the present invention;

[0018] Figure 2 This is an illustrative illustration of an embodiment of the present invention. Figure 1 A block diagram of the column reader in the Parquet reader;

[0019] Figure 3 This is an illustrative illustration of an embodiment of the present invention. Figure 2 A block diagram of a portion of the readers in the column reader;

[0020] Figure 4 This is an illustrative illustration based on an embodiment of the present invention. Figure 1 A block diagram of the record reconstructor in the Parquet reader;

[0021] Figure 5 This is an illustrative illustration based on an embodiment of the present invention. Figure 1A block diagram of the dictionary translation circuit in a Parquet reader; and

[0022] Figure 6 This is a flowchart illustrating a method for reading from a Parquet file for hardware implementation according to an embodiment of the present invention. Detailed Implementation

[0023] Overview

[0024] The embodiments of the invention described herein provide hardware-implemented readers and related methods for file formats such as Parquet. The disclosed techniques can be implemented, for example, as part of a microprocessor optimized for database applications, or in a standalone device.

[0025] Parquet files store data in a columnar format, initially represented as multiple records, each with multiple columns storing corresponding values. Records can have a nested structure, where values ​​can be defined as "required," "optional," or "repeated." For example, consider an example where each record stores a required first name, a required last name, optional child names, and optional grandchild names. In nested form, such a record could be written as follows:

[0026]

[0027] An alternative representation of this record is:

[0028] John Smith {{Tom,{Alice,Bob}},{Dick},{Harry,{Charlie,Dan,Edna}}

[0029] In the Parquet format, the nested structure of records that supports optional and repeating values ​​is represented by "repetition level" and "definition level". For example, the Parquet format, particularly the use of repetition level and definition level to represent nested record structures, is described in detail in the article "Dremel made simple with Parquet" in September 2013 and in Melnik et al.'s "Dremel: Interactive Analysis of Web-Scale Datasets" on pages 330-339 of the proceedings of the 36th International Conference on Web-Scale Datasets in 2010.

[0030] In Parquet format, the original table is divided into row groups, which are stored one after another in the file. The portion of a column belonging to a given row group is called a "column-chunk." The data for each row group is stored column-by-column in the file, making the data highly compressible. Each column-chunk is divided into pages, which are stored one after another. Each page includes a header, followed by the page's repetition level, then the page's definition level, and finally the page's value. The repetition level, definition level, and value are all compressed, for example, using Gzip or Snappy compression.

[0031] In some embodiments, a hardware-implemented reader is configured to read and decompress data from a Parquet file, and reconstruct selected records of the original table, without having to read and decompress the entire file. To this end, the reader is configured to read and decompress data in parallel from multiple different column blocks, and align data from different column blocks belonging to the same record. The reader uses this alignment to reconstruct the entire record based on the data read from each column block. In some embodiments, alignment is achieved using a backpressure mechanism that limits the rate at which data is read from different column blocks. In other embodiments, alignment is achieved by marking the read data with an identifier indicating the record to which the data belongs.

[0032] In some embodiments, the reader includes multiple hardware-implemented column readers operating in parallel. Each column reader is configured to be assigned to a corresponding selected column and to read and decompress repetition levels, definition levels, and values ​​from the selected column. The reader also includes a record reconstructor that aligns the data provided by the different column readers and reconstructs records based on the aligned data.

[0033] As described above, each Parquet page comprises three consecutive sections—a repeat level, a definition level, and a value. Typically, each column reader is configured to read and decompress the repeat level, definition level, and value in a mutually aligned manner. In one embodiment, a given column reader comprises three separate section readers configured to read and decompress the repeat level, definition level, and value in parallel, respectively. In another embodiment, a single section reader is configured to alternate between the repeat level, definition level, and value.

[0034] The disclosed Parquet reader can quickly and efficiently read data from Parquet files. The disclosed alignment mechanism allows the reader to reconstruct selected records while minimizing the reading and compression of unnecessary data. For example, unlike traditional reading methods, the disclosed Parquet reader does not require reading and decompressing the entire Parquet page to reconstruct specific values.

[0035] The disclosed reader can be configured to reconstruct and output a continuous stream of records, or one or more specific selected records. Various other reader features are also described, such as instant dictionary translation, operations on read values, and record reconstruction with controlled nesting levels.

[0036] While the embodiments described herein primarily relate to the Parquet format, the disclosed techniques can be used, by way of example, with other suitable file formats. For instance, repeating levels and definition levels are considered herein as an example of "structural information" used to specify the nested structure of records. In alternative embodiments, the disclosed techniques can be used to implement readers of other formats, where the structural information of records is represented in any other suitable manner. An example of an alternative file format is the Apache Optimized Rows and Columns (ORC) format.

[0037] System Description

[0038] Figure 1 This is a block diagram schematically illustrating a hardware-implemented Parquet reader 20 according to an embodiment of the present invention. Reader 20 is configured to read and reconstruct records from a Parquet file. Typically, reader 20 is implemented in hardware, such as an integrated circuit (IC) or a field-programmable gate array (FPGA). In some embodiments, although not mandatory, reader 20 is implemented as part of a microprocessor optimized for database applications. In some use cases, reader 20, for example, reconstructs and outputs one or more records from a Parquet file in response to a database query. In other use cases, reader 20 reconstructs and outputs a stream of records sequentially. Any other use cases may also be applied.

[0039] exist Figure 1 In the example, reader 20 accesses Parquet file 24 stored in dynamic random access memory (DRAM) 28. However, reader 20 can typically access Parquet file 24 stored on any other suitable memory or storage medium, or via a network, such as by communicating with a network interface controller (NIC).

[0040] Figure 1 The illustration at the bottom shows the structure of Parquet file 24. As mentioned above, file 24 consists of multiple line groups; each line group consists of multiple column blocks; and each column block consists of multiple pages. Each page includes a header, followed by the page's repetition level, then the page's definition level, and finally the page's value. File 24 ends with a "footer" section that includes file metadata.

[0041] In this example, reader 20 includes a memory interface 32 for accessing Parquet file 24 in memory 28. Reader 20 also includes multiple column reader circuits 36 (referred to as "column readers" for simplicity) and record reconstruction circuits 40 (referred to as "record reconstructors" for simplicity). Example implementations of column readers 36 and record reconstructors 40 are described below. Figure 2 and Figure 4 It was described in the text.

[0042] Typically, to reconstruct one or more records from Parquet file 24, each column reader 36 is associated with a corresponding column. The column reader 36 then reads data from the corresponding column, optionally decrypts, decompresses, and optionally decodes the data, and provides the decompressed data to record reconstructor 40. Record reconstructor 40 reconstructs the records based on the data provided by the column readers. Typically, the basic unit processed by each column reader 36 is a group of rows in a Parquet page.

[0043] Column reader and partial reader configuration

[0044] Figure 2 This is an illustrative illustration of an embodiment of the present invention. Figure 1 A block diagram of column readers 36 in Parquet reader 20. The individual column readers 36 of Parquet reader 20 typically have similar structures and are assigned different column blocks during operation.

[0045] In this example, column reader 36 includes a direct memory access (DMA) engine 44 for accessing DRAM 28 and multiple partial readers 48 (repetition-level reader 48A, definition-level reader 48B, and value reader 48C). The internal structure of the partial reader 48 according to the example embodiment is shown below. Figure 3 It was described in the text.

[0046] When column reader 36 reads a specific Parquet page belonging to its assigned column block, readers 48A, 48B, and 48C read and decompress the repeat level portion, definition level portion, and value portion of that page, respectively. Partial readers 48A and 48B are considered examples of one or more structure information readers configured to read and decompress structure information from columns assigned to column reader 36.

[0047] Typically, the compression mode used to compress the repeat level section, definition level section, and value section (e.g., Gzip or Snappy) has a variable compression rate. Therefore, the sizes of the repeat level section, definition level section, and value section (after compression) are variable and data-dependent. Consequently, the starting positions of the definition level section and value section in a Parquet page are also variable and data-dependent.

[0048] In various embodiments, the definition level reader 48B and value reader 48C can use various techniques to determine their starting positions in their respective sections within a given Parquet page. In one embodiment, the definition level reader 48B begins reading and decompressing data from the beginning of the Parquet page (which begins with the page title and repeating level sections), but discards data until it reaches the beginning of the definition level section. The purpose of this initial process is merely to identify the beginning of the definition level section. Only from this point does the definition level reader 48B begin outputting its decompressed data.

[0049] Similarly, value reader 48C starts from the beginning of the Parquet page and reads and decompresses the data in the repeating level and definition level sections to find the beginning of the value section. Value reader 48C discards the decompressed data until it reaches the beginning of the value section. Only from that point does value reader 48C begin to output its decompressed data.

[0050] In one embodiment, when processing a Parquet page begins, readers 48B and 48C perform an initial process of finding the starting position of their respective sections. Once all three section readers 48A-48C are ready, they begin reading and decompressing their respective sections in parallel to align (at least approximately align) the data they output. In this context, the term "alignment" means the association between a value and structural information about the definition of that value (e.g., repetition level and definition level).

[0051] In alternative embodiments, column reader 36 need not include three separate part readers 48 for reading the repeat level part, the definition level part, and the value part. For example, column reader 36 may include a single part reader 48 coupled to three storage buffers. In such an implementation, the part reader alternately reads and decompresses data from the repeat level part, the definition level part, and the value part. The part reader caches the decompressed data for each part in a corresponding buffer and typically holds three pointers to track the next position to be read from each part.

[0052] In another embodiment, the column reader may include a single part reader 48 that: (i) reads and decompresses corresponding data from the repeat level part and the definition level part into a buffer; then (ii) reads and decompresses corresponding data from the value part; and (iii) aligns the value with its repeat level and definition level.

[0053] Alternatively, column reader 36 can be implemented in any other manner and using any other suitable number of partial readers 48. In a typical implementation, the column reader reads, decompresses, and aligns the triples {repetition level, definition level, value} one by one, rather than reading and decompressing the entire Parquet page before aligning. In other words, the column reader typically reads, decompresses, and aligns a triple one before proceeding to read, decompress, and align the next triple.

[0054] exist Figure 2 In one embodiment, the column reader 36 also includes a microcontroller 60 configured to manage and control the operation of the column reader. In other tasks, the microcontroller 60 is configured to parse the page title of the Parquet page being read. The microcontroller 60 stores the page title in a buffer 64.

[0055] In some embodiments, the column reader 36 further includes a cyclic redundancy check (CRC) block coupled to the DMA engine 44. The CRC block verifies the CRC on the repeat-level data stream, the definition-level data stream, and the value data stream. The CRC result is provided to the microcontroller 60.

[0056] The column reader 36 also includes control logic 52. In other tasks, logic 52 is configured to trigger partial readers 48A, 48B, and 48C to provide corresponding repeat levels, definition levels, and values, as well as to align the repeat levels, definition levels, and values ​​provided by the partial readers. Logic 52 outputs values ​​to the record reconstructor 40 in parallel with the repeat levels provided by partial reader 48A.

[0057] In some embodiments, logic 52 is configured to perform dictionary translation of at least some values ​​before outputting values ​​to record reconstructor 40. The dictionary, or a portion thereof, is cached in dictionary cache 56 coupled to logic 52. Various aspects of dictionary translation will be discussed in further detail below.

[0058] Figure 3 This is an illustrative illustration of an embodiment of the present invention. Figure 2A block diagram of an example partial reader 48 in column reader 36. This configuration can be used to implement a repeating-level partial reader 48A, a definition-level partial reader 48B, and / or a value-level partial reader 48C. Partial reader 48 is configured to read data from a portion of a Parquet page (a repeating-level portion, a definition-level portion, or a value portion).

[0059] In this example, the partial reader 48 includes a configurable pipeline comprising (i) decryption logic 68 for decrypting data read from the partial reader, (ii) decompression logic 72 for decompressing the decrypted data, and (iii) a decoder 76 for decoding the decompressed data. A buffer 80 buffers the decrypted data between the decryption logic 68 and the decompression logic 72. A buffer 84 buffers the decompressed data between the decompression logic 72 and the decoder 76. Value processing logic 96 can be configured to execute a specified function on the decoded value, or otherwise manipulate the decoded value. An example of the function is given in the "Manipulation of Read Values" section below. Value processing logic 96 outputs the decoded value (to...) Figure 2 (Control logic 52 of column reader 36).

[0060] Partial reader 48 also includes a configuration register 92, which stores available configurations for decryption logic 68, decompression logic 72, and decoder 76. Configuration register 92 for decryption logic 68 can store, for example, a decryption key used to decrypt this part. Configuration register 92 for decompression logic 72 can store, for example, an identifier of the decompression mode to be used and any associated decompression parameters. Configuration register 92 for decoder 76 can store, for example, an identifier of the decoding mode to be used and any associated decoding parameters. Any suitable decryption, decompression, and decoding mode supported by Parquet can be used.

[0061] The partial reader 48 also includes control logic 88, which configures and controls various components of the partial reader.

[0062] One of the functions of buffers 80 and 84 is to apply back pressure to the preceding pipeline stages. The aspects of back pressure, and its use for data alignment between different sections of a Parquet page and between column blocks of different columns (or row groups) in a Parquet file, will be discussed further below.

[0063] In some embodiments, the partial reader 48 includes a dictionary buffer 100 coupled to the decoder 100. The buffer 100 may be used for storage, for example, for storing dictionary values ​​regarding repetition levels and definition levels.

[0064] Record Reconstruction

[0065] Figure 4 This is an illustrative illustration of an embodiment of the present invention. Figure 1 A block diagram of the record reconstructor 40 in the Parquet reader 20. The reconstructor 40 receives data from the individual column readers 36 and uses the data to reconstruct selected records of the Parquet file. The reconstructor 40 typically receives data from each column reader as three vectors—(i) a value vector, (ii) a repetition level vector, and (iii) a definition level vector.

[0066] The reconstructor 40 includes a programmable record reader 104 that reconstructs records based on data provided by the column reader. The reconstructed records are provided as output of the Parquet reader 20. The reconstructor 40 also includes a pattern memory 120 that stores the patterns of the Parquet files being read.

[0067] In some embodiments, record reconstructor 40 may include dictionary circuitry 112 and dictionary cache 108, the dictionary circuitry 112 performing dictionary translation of values ​​and the dictionary cache 108 storing dictionary entries. However, this is not mandatory—in some embodiments, all dictionary translation operations, including caching and cache lookups, are performed in column reader 36. Dictionary translation features, including various options for the “division of labor” between column reader and record reconstructor, will be described in further detail below.

[0068] In some embodiments, the record reconstructor 40 further includes function circuitry 116 that performs a predefined function on the reconstructed record before outputting the record. Some functions control the level of nesting of the record being reconstructed by the record reconstructor 40. Consider the example record described in the overview section above, which has two levels of nesting:

[0069] name surname Children & grandchildren John Smith {{Tom, {Alice, Bob}}, {Dick}, {Harry, {Charlie, Dan, Edna}}

[0070] This record can also be represented as follows, still in a fully nested format:

[0071]

[0072] Decomposing a nested level will produce three records:

[0073]

[0074] John Smith Dick -

[0075]

[0076] Decomposing two nesting levels will produce six plane (non-nested) records:

[0077] John Smith Tom Alice

[0078] John Smith Tom Bob

[0079] John Smith Dick Charlie

[0080] John Smith Harry Dan

[0081] John Smith Harry Edna

[0082] John Smith Harry

[0083] In some embodiments, function circuit 116 is configured to specify the number of levels to be decomposed in a reconstructed record, for example, based on each record or as a general operating mode. The number ranges from zero to the maximum nested level.

[0084] In some embodiments, the programmable record reader 104 in the reconstructor 40 includes multiple processing engines (e.g., kernels or processors) configured to reconstruct multiple corresponding records simultaneously. The reader 104 may include a scheduler that flexibly allocates records to the processing engines.

[0085] In some embodiments, record reconstructor 40 is configured to reconstruct multiple record streams in parallel or alternately (e.g., using different processing engines simultaneously). Each record stream typically originates from a different query and therefore typically has a separate definition of the columns to be read, the functions to be applied, etc. In some embodiments, record reader 104 in reconstructor 40 is configured to apply backpressure to column reader 36 only for one or more selected record streams and avoid applying backpressure to other streams.

[0086] When reconstructing multiple recording streams alternately, once all decompression buffers 84 ( Figure 3 If the decompression buffers are empty (at the end of each line group), reader 104 can switch between recording streams. Alternatively, reader 104 can switch between recording streams by saving the current state of the decompression buffers (“context”) or by repeating decompression when the same recording stream is rescheduled for reconstruction, while some decompression buffers are not empty. In some embodiments, recording reader 104 in reconstructor 40 is configured to use context switching only for one or more selected recording streams and wait for other streams until the end of the line group. In any case, pausing or resuming one stream will not affect other streams.

[0087] In various embodiments, reader 20 can apply either cooperative context switching or preemptive context switching between record streams. In cooperative context switching, the external controller allows reader 20 to complete the reconstruction of the entire row group and then configures the reader only to reconstruct the row group for different contexts (typically different queries related to different sets of columns). In preemptive context switching, if reader 20 does not complete the reconstruction of the row group after a predefined time period, the controller instructs reader 20 to stop in an intermediate row group. The controller saves the context of the intermediate row group state for the specified reader (e.g., the page index for each column reader and the value index in each page) and configures the reader to operate on another row group. The controller can later configure the reader to resume reading the preempted row group from its preempted position.

[0088] In various embodiments, the record reconstructor 40 can output the reconstructed record to any suitable "user system," such as a processor that applies further processing to the record. In some embodiments, the reconstructor 40 is capable of receiving and acting upon reverse pressure signaling (e.g., signals or notifications) from the user system. Typically, the user system sends a "pause" notification when it is unable to receive new records and a "resume" notification when it is able to receive records again. Upon receiving a "pause" notification, the record reconstructor 40 can begin buffering the reconstructed record, provided space permits. Otherwise, the record reconstructor can propagate the reverse pressure forward to the column reader.

[0089] In some embodiments, the record reconstructor 40 outputs various types of metadata along with the reconstructed record. Metadata may include, for example, additional information related to a specific field, such as a Boolean value indicating whether a field value satisfies a condition, a numerical value indicating the first letter of a string field, etc. As another example, metadata may indicate nesting levels (or other structural information) in a manner different from that specified in the Parquet format. For example, metadata may indicate whether nested values ​​in a field are empty.

[0090] Additional embodiments and variations

[0091] Align using reverse pressure

[0092] In some embodiments, the record reconstructor 40 and the column reader 36 (including various section readers 48) use a reverse pressure mechanism to control the rate at which data is read from various sections of each column block. For example, this rate control is important for maintaining alignment between data read from different column blocks.

[0093] In the example implementation, under the control of the record reader 104 of the record refactorer 40, each column reader 36 operates independently of the other column readers. The record reader 104 assigns a corresponding column block to each column reader and instructs the column reader to read which Parquet pages from the column block.

[0094] Because each column reader 36 operates independently, and because the data size and compression rate are variable, some column readers can return data at a higher rate than others. However, the record reader 104 is designed to maintain alignment between the column readers 36, that is, to ensure that the data provided by different column readers at a given time corresponds to the same record or a small number of adjacent records.

[0095] In an example embodiment, record reader 104 can detect that, at a given time, the data provided by each column reader belongs to more than a predefined maximum number of adjacent records. In response to detecting this misalignment, record reader 104 can temporarily pause one or more of the relatively faster column readers to allow one or more slower column readers to catch up. When alignment is restored, i.e., when the data provided by each column reader belongs to no more than the predefined maximum number of adjacent records, record reader 104 can resume the paused column readers.

[0096] A typical example of alignment requirements is when one column (denoted as col1) stores an integer field, while another column (denoted as col2) holds a corresponding list. In this case, for each record, the column reader assigned to col1 needs to read a single value, while the column reader assigned to col2 needs to read the entire list.

[0097] Typically, each column reader 36 internally transmits reverse pressure to the respective section readers 48 and within the pipeline of each section reader 48. For example, when a particular column reader 36 is paused by the recording reader 104, the decoder 76 of the section reader 48 is paused, and when it is resumed, the decoder is resumed. In a given section reader 48 (see...) Figure 3 Within the buffer, when decoder 76 is paused, buffer 84 will gradually fill with decompressed data. When a certain padding level is exceeded, buffer 84 will pause decompression logic 72. Pausing decompression logic 72 will cause buffer 80 to gradually fill with decrypted data. When a certain padding level is exceeded, buffer 80 will pause decryption logic 68.

[0098] When the column reader resumes operation, the resumption is similarly propagated internally to the partial readers—each buffer (84 and 80) begins to empty, and the preceding pipeline stage resumes when sufficiently empty. In various embodiments, record reader 104 and column reader 36 (including partial reader 48 and its internal buffers and pipeline stages) can use any suitable signaling to apply reverse pressure. For example, an "ready" signal between stages can cancel an assertion to pause, and cancel an assertion to resume operation.

[0099] Record reconstructor 40 typically uses available information about the document being read, such as document structure information and / or patterns, to maintain alignment between values ​​provided by the respective column readers 36. In some embodiments, the record reconstructor determines the amount of data each record needs to obtain from each column reader and maintains alignment between column readers by obtaining an appropriate amount of data from each column reader.

[0100] For example, if none of the columns being read have any nesting, the record reconstructor 40 simply retrieves the value from each column. If a field is wide (wider than the bus size), the record reconstructor may need to receive that field from the column reader over multiple cycles per record, whereas for other fields, only one cycle is needed per field. For variable-size fields, such as strings, the record reconstructor can check the field's length, which is typically specified in the first few bytes of the field. The record reconstructor can then use the length information to set the number of cycles required to receive the field from the column reader. When a column has nesting, structural information (e.g., repetition level and definition level in Parquet) helps the record reconstructor determine how to align that column with other columns and reconstruct the record.

[0101] In some cases, to reconstruct a record, the number of columns requested by Parquet reader 20 is greater than the number of column readers 36. In some embodiments, record reconstructor 40 handles such requests by reconstructing a partial record based on a subset of columns not exceeding the number of column readers, and saves the partial record to memory. The collection of partial records can then be merged (e.g., via an external controller or user system) to form a fully reconstructed record with any desired number of columns.

[0102] Dictionary translation

[0103] According to the Parquet format, a given column block may optionally contain a dictionary. The dictionary represents common values ​​(e.g., strings) with shorter keys. Subsequent Parquet pages within the column block include the short keys instead of the longer actual values, which further reduces the file size. Dictionaries can be used to represent strings and / or any other supported value types, such as double or long. Dictionaries can represent values ​​with a constant size (i.e., all values ​​in the dictionary have the same size) or values ​​with a variable size (i.e., different values ​​in the dictionary can differ in size from each other). Dictionaries are typically placed at the beginning of the column block, before the first Parquet page, and are often compressed using, for example, Gzip or Snappy.

[0104] In some embodiments, the Parquet reader 20 is configured to read, decompress, and store one or more dictionaries from one or more column blocks to translate the keys read from the column blocks into actual values ​​on the fly, and to reconstruct and output records with actual values.

[0105] In this embodiment, when a dictionary is encountered, reader 20 uses one of the partial readers 48 to decompress the dictionary and stores the decompressed dictionary in a nearby memory, such as static random access memory (SRAM). When the key is later translated into an actual value, the circuitry in reader 20 uses the key to derive the memory address where the corresponding actual value is stored.

[0106] Figure 5 This is a schematic illustration based on an embodiment of the present invention. Figure 1 A block diagram of the dictionary translation circuit 122 in the Parquet reader 20. Circuit 122 can be implemented, for example, as logic 52 and dictionary cache 56 in the column reader 36. Figure 2 Part of the dictionary circuit 112 in the record reconstructor 40. Figure 4 It can be part of, or implemented in, both of the dictionary. As another example, under the assumption that not all column readers will access the dictionary simultaneously, appropriate memory can be used to share the dictionary's storage among two or more column readers. In yet another example, the dictionary is stored in a memory subsystem, and the relevant portion of the memory is cached in a nearby close-by cache memory.

[0107] Dictionary circuit 122 receives keys read from a Parquet page as input and outputs values ​​represented by the keys according to the dictionary. Circuit 122 includes an indirect addressing table 124 and a map 128. Map 128 stores the actual values ​​and is accessed via addresses (represented as offsets from some base address). Table 124 and map 128 are referred to herein as "dictionary data structures". In alternative embodiments, any other suitable type of dictionary data structure may be used.

[0108] For a dictionary of constant-size values, circuit 122 directly calculates the offset based on the key, for example, by multiplying the key by the constant size of the value. This offset is then used to access map 128 and extract the value from it. In this example, indirect addressing table 124 is not used. For a dictionary of variable-size values, circuit 122 uses the key to access indirect addressing table 124 and extracts the offset and size from it. The offset and size are then used to access map 128 and extract the value from it. In an alternative embodiment, indirect addressing table 124 can also be used to access a dictionary of constant-size values ​​without using multiplication.

[0109] Typically, circuit 122 constructs a dictionary (e.g., filling mapping 128 and indirect addressing table 124, if used) when reading a dictionary from a column block. Later, when reading a Parquet page from a column block, circuit 122 performs an on-the-fly translation using the filled mapping 128 and the filled table 124, if used.

[0110] Operations on read values

[0111] In some embodiments, the Parquet reader 20 is configured to modify values ​​read from a Parquet file and insert the modified values ​​into the reconstructed record. Typically, an operation is specified for a selected field (selected column). This operation can be performed at any stage of the read and reconstruction process, such as before record reconstruction, or after record reconstruction and before outputting the reconstructed record.

[0112] For example, it can be done in column reader 36 (e.g., via...) Figure 3 The value processing logic 96 in the partial reader 48 or in the record reconstructor 40 (e.g., via...) Figure 4 The function circuit 116) performs value operations. Operations performed in the column reader are typically (though not necessarily) simple, hardware-controlled operations, such as comparisons of values ​​with thresholds or simple logic functions. Operations performed in the record reconstructor can be more complex, software-controlled operations, such as operations derived from queries that trigger read operations.

[0113] Reader 20 can perform any suitable type of operation on the value. Several illustrative, non-restrictive examples of operations include the following (regarding the field (column) denoted as "field1"):

[0114] Apply functions to the values ​​of fields, such as Round(field1), field1*4, Sign(field1), etc.

[0115] Output a boolean flag ("0" or "1") instead of field1, depending on whether the condition defined on field1 is true or false, for example, "field1>6". Another example of a boolean function is the following function: returns "1" if the specified value exists in the list, otherwise returns "0".

[0116] Manipulate nested values, such as modifying a list or a selected item on the entire list, or apply functions to a list, such as finding the maximum, minimum, or sum of values ​​in the list.

[0117] Perform multiple operations on a single field, for example, output multiple boolean flags in response to multiple conditions defined on the same field (e.g., "field1>7", "field1<30", "field1>100").

[0118] Perform various functions on the string, such as logical functions, like col1 == "the" or col1 == "%the", where % is a wildcard.

[0119] Perform various functions on date types, such as functions to extract the year and month from a date field in col4.

[0120] For example, by combining operations on multiple fields, records can be filtered (i.e., selectively discarded or retained) based on criteria defined for the values ​​of one or more fields. Filtering can be performed before or after reconstructing records.

[0121] In various embodiments, reader 20 can perform various filtering operations to determine which records to output and which to discard. In some embodiments, the filtering procedure is based on a single column, such as col1 > 7. In this case, record reconstructor 40 outputs only records that meet the specified criteria and discards records that do not meet the specified criteria. Note that the output is a complete record (e.g., col1, col2, ..., col8), even if the criteria only relate to a specific column.

[0122] In another embodiment, the filter is based on a combination of two or more columns, such as "(Col1>7 and col2==“The record”) or (extract_date(col3)-current_date<50)". In this embodiment, the columns in the filter definition can be of different types. Furthermore, various operations can be performed on the columns to reach a final decision about whether to output or discard a particular record.

[0123] In some cases, the filtering procedure is not necessarily related to the records output from the reconstructor 40. Additionally or alternatively, the records being output may differ from the original records in the file. For example, consider the filtering procedure "(Col1>17andcol2 in(“The record”,“The sum”,“The Best”))or(extract_date(col3)-current_date<10)". In one embodiment, the record reconstructor 40 checks whether a record meets a condition. After identifying records that meet the condition, the record reconstructor 40 outputs the corresponding records for different columns, such as "col1-5,col2,(if col2==“TheBest”,1,0),extract_date(col3),extract_month(col3),col4,col5,col6,7*col7-43,is_Null(col8)".

[0124] The filtering procedure above is given purely as an example to demonstrate the capabilities provided by filtering in reader 20. In alternative embodiments, any other suitable filtering procedure may be used.

[0125] Typically, when deriving the post-operation value from one or more fields of a record (e.g., when performing a function on one or more fields), reader 20 outputs the post-operation value in the metadata accompanying the reconstructed record. In some embodiments, reader 20 outputs the post-operation value instead of the field values ​​read from the column. In other embodiments, reader 20 outputs both the post-operation value and the values ​​read from the column. This is typically accomplished by outputting additional fields.

[0126] In some embodiments, although the reader 20 is typically configured based on the file being read, the record refactorer 40 may be configured to modify values ​​based on the processing (e.g., the query being processed) rather than the file, and thus further optimize the processing.

[0127] Overall Methodology Description

[0128] Figure 6This is a flowchart illustrating, according to an embodiment of the invention, a hardware-implemented method for reading from a Parquet file, executed by a Parquet reader 20. The method begins in column specification step 130, where the record reconstructor 40, for example in response to a query, specifies the set of columns to be read from the Parquet file. In configuration step 134, the record reconstructor 40 assigns a corresponding column reader 36 to each column to be read and configures the column reader 36 accordingly.

[0129] The column reader 36, assigned to that column, performs the following steps (138-146) on each Parquet page for each column block being read. In partial initialization step 138, column reader 36 initializes partial reader 48 with the starting position from which the repeat level portion, definition level portion, and value portion will be read in partial initialization step 138. In readout step 142, partial reader 48 reads, decrypts, decompresses, and decodes the repeat level, definition level, and value from the Parquet page. Column reader 36 outputs the repeat level, definition level, and value to record reconstructor 40.

[0130] In record reconstruction step 150, record reconstructor 150 reconstructs one or more records based on the repetition level, definition level, and value provided by the respective column readers 36.

[0131] like Figure 1-5 The configuration of the Parquet reader 20 and its components (e.g., column reader 36, section reader 48, and record reconstructor 40) shown is an example configuration depicted purely for conceptual clarity. In alternative embodiments, any other suitable configuration may be used.

[0132] Parquet reader 20 and its components (such as column reader 36, section reader 48, and record reconstructor 40) can be implemented using any suitable hardware (e.g., in application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0133] In some embodiments, specific portions and components of the Parquet reader 20 (e.g., column reader 36, section reader 48, and record reconstructor 40) may be implemented using a programmable processor programmed in software to perform the functions described herein. The software may be downloaded electronically to the processor via a network, or alternatively or additionally, the software may be provided and / or stored on a non-transitory tangible medium (such as magnetic storage, optical storage, or electronic storage).

[0134] It should be understood that the above embodiments are cited by way of example, and the invention is not limited to what has been specifically shown and described above. Rather, the scope of the invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications of the invention that would be conceived by one of skill in the art upon reading the above description and that are not disclosed in the prior art. Documents incorporated herein by reference are considered part of this application, and the definitions in this specification should be considered only, unless any terms are defined in these incorporated documents in a manner that conflicts to some extent with the definitions expressly or implicitly made in this specification.

Claims

1. A hardware-implemented document reader, comprising: An interface configured to access a file comprising multiple records, wherein the records store values ​​according to a nested structure supporting optional and repeating values, and wherein the file is stored in a column format with multiple columns, each column storing (i) a compressed value and (ii) corresponding compressed structure information that associates the value in the column with the nested structure of the records; Multiple hardware-implemented column readers, each configured to be assigned to a corresponding selected column, and to read and decompress the values ​​and structural information from at least a portion of the selected column; and A hardware-implemented record reconstructor is configured to align the column readers with each other to provide corresponding outputs for one or more selected records belonging to the file, to reconstruct the one or more selected records based on at least a portion of the columns read by the aligned column readers, and to output the reconstructed records.

2. The document reader according to claim 1, wherein, The column format includes the Parquet format, and the structure information includes the repetition level and definition level of the value.

3. The document reader according to claim 1, wherein, The record reconstructor is configured to apply reverse pressure to one or more of the column readers in order to align the corresponding outputs of the column readers to belong to no more than a predefined number of adjacent records.

4. The document reader according to claim 1, wherein, The record reconstructor is configured to determine the appropriate data size that each record needs to obtain from each column reader, and to align the column readers by obtaining the determined data size from each column reader.

5. The document reader according to claim 1, wherein, The given column reader is configured to align at least some of the decompressed values ​​with the corresponding decompressed structural information before reading and decompressing subsequent values ​​and subsequent structural information from the selected column.

6. The document reader according to any one of claims 1-5, wherein, A given column reader includes a value reader configured to read and decompress the values ​​of the selected column, and one or more structure information readers configured to read and decompress the structure information of the selected column.

7. The document reader according to claim 6, wherein, The structural information includes the repetition level and definition level of the value, and wherein the structural information reader includes a repetition level reader configured to read and decompress the repetition level, and a definition level reader configured to read and decompress the definition level.

8. The document reader according to any one of claims 1-5, wherein, A given column reader comprises a single reader configured to alternately read and decompress both the values ​​of the selected column and the structural information of the selected column.

9. The document reader according to any one of claims 1-5, wherein, When reconstructing nested records, the record reconstructor is configured to decompose one or more nesting levels of the nested record, thereby outputting multiple records to replace the nested record.

10. The document reader according to any one of claims 1-5, wherein, In response to a request to reconstruct a set of columns greater than the number of columns read by the column reader, the record reconstructor is configured to reconstruct and output two or more partial record sets, each corresponding to a subset of the requested set of columns.

11. The document reader according to any one of claims 1-5, further comprising a hardware-implemented dictionary circuit configured to read a dictionary of values ​​represented by corresponding keys from the document, and subsequently translate the keys read from the document into corresponding values ​​so as to place the translated values ​​in the reconstructed record.

12. The file reader of claim 11, wherein the file reader comprises a dictionary data structure that maps the keys to corresponding values, wherein, The dictionary circuit is configured to populate the dictionary data structure when reading the dictionary from the file.

13. The document reader according to claim 11, wherein, The dictionary circuit is configured to store a portion of the dictionary in a cache.

14. The document reader according to any one of claims 1-5, wherein, One or more of the column readers and the record reconstructor are configured to modify one or more of the values ​​read from the file.

15. The document reader according to claim 14, wherein, The record reconstructor is configured to output only the modified value, or output both the value read from the file and the modified value.

16. The document reader according to claim 14, wherein, The record reconstructor is configured to specify the modification of the value based on the received query.

17. The document reader according to any one of claims 1-5, wherein, The record reconstructor is configured to filter the record based on one or both of the following: (i) criteria defined for one or more of the values, and (ii) a received query.

18. The document reader according to any one of claims 1-5, wherein, The record reconstructor includes multiple processing engines configured to reconstruct multiple corresponding records simultaneously.

19. The document reader according to any one of claims 1-5, wherein, The record reconstructor is configured to reconstruct multiple record streams in parallel.

20. The document reader according to claim 19, wherein, The record reconstructor is configured to reconstruct the multiple record streams independently of each other.

21. The document reader according to any one of claims 1-5, wherein, The record reconstructor is configured to apply reverse pressure to the column reader only for a selected subset of the record stream.

22. A method for hardware-implemented file reading, comprising: A column reader implemented using multiple hardware accesses a file, wherein the file comprises multiple records, wherein the records store values ​​according to a nested structure that supports optional and repeating values, and wherein the file is stored in a column format having multiple columns, each column storing (i) compressed values ​​and (ii) corresponding compressed structure information that associates the values ​​in the column with the nested structure of the records; Each column reader is assigned to a corresponding selected column, and the values ​​and structural information are read and decompressed from at least a portion of the selected columns; and A hardware-implemented record reconstructor aligns the column readers to provide corresponding outputs for one or more selected records belonging to the file, reconstructs the one or more selected records based on at least a portion of the columns read by the column readers, and outputs the reconstructed records.

23. The method according to claim 22, wherein, The column format includes the Parquet format, and the structure information includes the repetition level and definition level of the value.

24. The method according to claim 22, wherein, Reconstructing the record involves applying reverse pressure to one or more of the column readers to align the corresponding outputs of the column readers to belong to no more than a predefined number of adjacent records.

25. The method according to claim 22, wherein, Reconstructing the records includes: determining the appropriate data size that each record needs to obtain from each column reader, and aligning the column readers by obtaining the determined data size from each column reader.

26. The method according to claim 22, wherein, Reading and decompressing the values ​​and the structural information includes: in a given column reader, aligning at least some of the decompressed values ​​with the corresponding decompressed structural information before reading and decompressing subsequent values ​​and subsequent structural information from the selected column.

27. The method according to any one of claims 22-26, wherein, Reading and decompressing the values ​​and the structure information includes: in a given column reader, reading and decompressing the values ​​of the selected columns by a value reader, and reading and decompressing the structure information of the selected columns by one or more structure information readers.

28. The method according to claim 27, wherein, The structural information includes the repetition level and definition level of the value, and wherein reading and decompressing the structural information includes reading and decompressing the repetition level by a repetition level reader and reading and decompressing the definition level by a definition level reader.

29. The method according to any one of claims 22-26, wherein, Reading and decompressing the value and the structure information includes: in a given column reader, alternatingly reading and decompressing both the value and the structure information of the selected column by a single reader.

30. The method according to any one of claims 22-26, wherein, Reconstructing the record includes: when reconstructing a nested record, decomposing one or more nesting levels of the nested record, thereby outputting multiple records to replace the nested record.

31. The method according to any one of claims 22-26, wherein, Reconstructing the records includes, in response to a request to reconstruct a set of columns greater than the number of columns in the column reader, reconstructing and outputting two or more partial record sets, each partial record set corresponding to a subset of the requested set of columns.

32. The method according to any one of claims 22-26, further comprising using a hardware-implemented dictionary circuit to read a dictionary of values ​​represented by corresponding keys from the file, and subsequently translating the keys read from the file into corresponding values ​​so as to place the translated values ​​in the reconstructed record.

33. The method according to claim 32, wherein, Reading the dictionary includes filling a dictionary data structure that maps the keys to corresponding values, and wherein translating the keys to corresponding values ​​includes mapping the keys to corresponding values ​​through the filled dictionary data structure.

34. The method of claim 32, wherein the method includes storing a portion of the dictionary in a cache.

35. The method according to any one of claims 22-26, wherein the method includes modifying one or more of the values ​​read from the file.

36. The method according to claim 35, wherein, The output of the reconstructed record may include outputting only the modified value, or outputting both the value read from the file and the modified value.

37. The method of claim 35, wherein the method includes specifying a modification of the value based on a received query.

38. The method according to any one of claims 22-26, wherein, Reconstructing the record includes filtering the record based on one or both of the following: (i) criteria defined for one or more of the values, and (ii) the received query.

39. The method according to any one of claims 22-26, wherein, Reconstructing the records involves applying multiple processing engines to simultaneously reconstruct multiple corresponding records.

40. The method according to any one of claims 22-26, wherein, Reconstructing the records involves reconstructing multiple record streams in parallel.

41. The method according to claim 40, wherein, Reconstructing the records involves reconstructing the multiple record streams independently of each other.

42. The method according to any one of claims 22-26, wherein, Reconstructing the records involves applying reverse pressure to the column reader only for a selected subset of the record stream.

Images