File reader implemented by hardware

Abstract

A hardware-implemented file reader (20) includes an interface (32), a plurality of hardware-implemented column readers (36), and a record reconstructor (40). The interface is configured to access a file (24) comprising a plurality of records. The recording stores values according to a nested structure that supports selectable values and duplicate values. The file is stored in a column format having a plurality of columns, each column storing (i) a compressed value and (ii) corresponding compressed structure information associating values in the column with a recorded nested structure. Each column reader is configured to be assigned to a respective selected column, and to read and decompress values and structure information from at least a portion of the selected column. The record reconstructor is configured to reconstruct one or more of the records according to at least a portion of the column read by the column reader, and output the reconstructed records.

Description

[0001] This application is a divisional application of application No. 202180062362.1, filed on September 12, 2021, having the title “Hardware-Implemented File Reader”. TECHNICAL FIELD

[0002] The present invention relates generally to digital hardware and microprocessor design, and in particular to methods and systems for hardware-implemented file reading. BACKGROUND

[0003] Apache Parquet is a popular columnar data storage format. Parquet files have a complex hierarchy and support multiple compression and encoding modes. There have been some attempts to process Parquet files in hardware. For example, Van Leeuwen et al. describe a Parquet-to-Arrow converter in “High-throughput conversion of Apache Parquet files to Apache Arrow in-memory format using FPGAs”, CompSys 2019, June 4, 2019. SUMMARY

[0004] Embodiments of the invention described herein provide a hardware-implemented file reader that includes an interface, a plurality of hardware-implemented column readers, and a hardware-implemented record reconstituter. The interface is configured to access a file that includes a plurality of records. The records store values according to a nested structure that supports optional values and repeated values. The file is stored in a columnar format having a plurality of columns, each column storing (i) compressed values and (ii) corresponding compressed structure information that associates the values in the column with the nested structure of the records. Each column reader is configured to be assigned to a respective selected column and to read and decompress values and structure information from at least a portion of the selected column. The record reconstituter is configured to reconstitute one or more of the records from at least the portion of the column read by the column readers and to output the reconstituted records.

[0005] In some embodiments, the columnar format is a Parquet format and the structure information includes repetition levels and definition levels of the values. In one embodiment, the record reconstituter is configured to apply backpressure to one or more of the column readers in order to align respective outputs of the column readers to belong to no more than a predefined number of adjacent records.

[0006] In example embodiments, the record reconfigurator is configured to determine a respective data size that each record needs to obtain from each column reader, and to maintain alignment between column readers by obtaining the determined data size from each column reader. In disclosed embodiments, a given column reader is configured to align at least some decompressed values with corresponding decompressed structure information before reading and decompressing subsequent values and subsequent structure information from a selected column.

[0007] In some embodiments, a given column reader includes a value reader configured to read and decompress values of a selected column, and one or more structure information readers configured to read and decompress structure information of the selected column. In example embodiments, the structure information includes a repetition level and a definition level of the values, and the structure information readers include a repetition level reader configured to read and decompress the repetition level, and a definition level reader configured to read and decompress the definition level. In another embodiment, a given column reader includes a single reader configured to alternately read and decompress both values of a selected column and structure information of the selected column.

[0008] In yet another embodiment, in reconstructing a nested record, the record reconfigurator is configured to decompose one or more nesting levels of the nested record, thereby outputting a plurality of records in place of the nested record. In still another embodiment, in response to a request to reconstruct a set of columns larger than a number of column readers, the record reconfigurator is configured to reconstruct and output two or more partial record sets, each corresponding to a respective subset of the requested set of columns.

[0009] In some embodiments, the file reader further includes a hardware-implemented dictionary circuit configured to read a dictionary of some values represented by respective keys from the file, and subsequently translate the keys read from the file into corresponding values, so as to place the translated values in the reconstructed records. In one embodiment, the file reader includes a dictionary data structure that maps keys to respective values, and the dictionary circuit is configured to populate the dictionary data structure when reading the dictionary from the file. In one embodiment, the dictionary circuit is configured to keep a portion of the dictionary in a cache.

[0010] In some embodiments, one or more of the column readers and the record reconfigurator are configured to modify one or more values read from the file. In example embodiments, the record reconfigurator is configured to output only the modified values, or to output both the values read from the file and the modified values. In one embodiment, the record reconfigurator is configured to specify the modification of values based on a received query.

[0011] In another embodiment, the record reconfigurator is configured to filter records based on one or both of (i) a defined criterion on one or more of the values, and (ii) a received query. In a disclosed embodiment, the record reconfigurator includes a plurality of processing engines configured to reconfigure a plurality of respective records concurrently.

[0012] In some embodiments, the record reconfigurator is configured to reconfigure a plurality of record streams in parallel. In one embodiment, the record reconfigurator is configured to reconfigure a plurality of record streams independently of one another. In a disclosed embodiment, the record reconfigurator is configured to apply back pressure to the column readers only for a selected subset of the record streams.

[0013] According to embodiments of the present invention, there is also provided a method for hardware-implemented file readout. The method includes accessing a file using a plurality of hardware-implemented column readers. The file includes a plurality of records, and the records store values according to a nested structure that supports optional values and repeated values. The file is stored in a column format having a plurality of columns, each column storing (i) compressed values and (ii) corresponding compressed structure information that associates the values in the column with the nested structure of the records. Each column reader is assigned to a respective selected column, and reads and decompresses both the values and the structure information from at least a portion of the selected column. One or more of the records are reconfigured according to at least the portion of the column read by the column reader, using a hardware-implemented record reconfigurator, and the reconfigured records are output.

[0014] The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a block diagram schematically illustrating a hardware-implemented Parquet reader according to embodiments of the present invention; Figure 2 is a block diagram schematically illustrating a column reader in the Parquet reader of Figure 1 according to embodiments of the present invention; Figure 3 is a block diagram schematically illustrating a partial reader in the column reader of Figure 2 according to embodiments of the present invention; Figure 4 is a block diagram schematically illustrating a record reconfigurator in the Parquet reader of Figure 1 according to embodiments of the present invention; Figure 5 is a block diagram schematically illustrating a dictionary translation circuit in the Parquet reader of Figure 1 according to embodiments of the present invention; and Figure 6is a flowchart schematically showing a method for reading out from a Parquet file according to an embodiment of the application. DETAILED DESCRIPTION

[0016] SUMMARY Embodiments of the application described herein provide a hardware-implemented reader for file formats such as Parquet and related methods. The disclosed technology can be implemented, for example, as part of a microprocessor optimized for database applications, or in a standalone device.

[0017] A Parquet file stores data originally represented as a plurality of records, each record having a plurality of columns storing respective values, in columnar format. Records can have a nested structure, in which values can be defined as "required", "optional", or "repeated". For example, consider an example in which each record stores a required first name, a required last name, an optional child name, and an optional grandchild name. In nested form, such a record can be written as follows:

[0018] An alternative representation of such a record is:

[0019] In the Parquet format, the nested structure of records supporting optional values and repeated values is represented by "repetition levels" and "definition levels". The Parquet format, and in particular the use of repetition levels and definition levels to represent nested record structures, is described in detail in an article entitled "Dremel made simple with Parquet", September 2013, and in Melnik et al., "Dremel: Interactive Analysis of Web-Scale Datasets", Proceedings of the 36th SIGMOD International Conference on Management of Data, pages 330-339, 2010.

[0020] According to the Parquet format, the original table is divided into row groups, which are stored one after the other in the file. The portion of a column belonging to a given row group is called a "column-chunk". The data of each row group is stored in the file column-chunk by column-chunk, so that the data has a high degree of compressibility. Each column-chunk is divided into pages, which are stored one after the other. Each page includes a header, followed by the repetition levels of the page, followed by the definition levels of the page, followed by the values of the page. The repetition levels, the definition levels, and the values are compressed, for example, using Gzip or Snappy compression.

[0021] In some embodiments, the hardware-implemented reader is configured to read and decompress data from a Parquet file, and to reconstruct selected records of the original table without having to read and decompress the entire file. To this end, the reader is configured to read and decompress data from multiple different column chunks in parallel, and to align data from different column chunks that belong to the same record. The reader uses this alignment to reconstruct entire records from the data read from the various column chunks. In some embodiments, the alignment is achieved using a backpressure mechanism that limits the rate at which data is read out from different column chunks. In other embodiments, the alignment is achieved by marking the read data with an identifier that indicates the record to which the data belongs.

[0022] In some embodiments, the reader includes multiple hardware-implemented column readers that operate in parallel. Each column reader is configured to be assigned to a respective selected column, and to read and decompress the repetition level, definition level, and value from the selected column. The reader also includes a record reconstructor that aligns the data provided by the different column readers, and reconstructs records from the aligned data.

[0023] As noted above, each Parquet page includes three consecutive sections - the repetition level, the definition level, and the value. Typically, each column reader is configured to read and decompress the repetition level, the definition level, and the value in a mutually aligned manner. In one embodiment, a given column reader includes three separate section readers that are configured to read and decompress the repetition level, the definition level, and the value, respectively, in parallel. In another embodiment, a single section reader is configured to alternate between the repetition level, the definition level, and the value.

[0024] The disclosed Parquet reader is able to read out data from a Parquet file quickly and efficiently. The disclosed alignment mechanism enables the reader to reconstruct selected records while minimizing the readout and decompression of unnecessary data. For example, unlike conventional readout patterns, the disclosed Parquet reader does not need to read and decompress an entire Parquet page in order to reconstruct a particular value.

[0025] The disclosed reader can be configured to reconstruct and output a continuous stream of records, or one or more particular selected records. Various other reader features are also described, such as on-the-fly dictionary translation, read value operations, and record reconstruction with controlled nesting levels.

[0026] While the embodiments described herein primarily relate to the Parquet format, the disclosed techniques can be used with other suitable file formats as examples. For example, the repetition level and definition level are treated herein as one example of "structure information" used to specify the nested structure of records. In alternative embodiments, the disclosed techniques can be used to implement readers for other formats in which the structure information for records is represented in any other suitable manner. One example of an alternative file format is the Apache Optimized Row Column (ORC) format.

[0027] System Description Figure 1 is a block diagram schematically showing a hardware implementation of a Parquet reader 20 according to an embodiment of the invention. The reader 20 is configured to read and reconstruct records from a Parquet file. Typically, the reader 20 is implemented in hardware, for example in an integrated circuit (IC) or a field programmable gate array (FPGA). In some embodiments, although not necessarily, the reader 20 is implemented as part of a microprocessor optimized for database applications. In some use cases, the reader 20 reconstructs and outputs one or more records from a Parquet file, for example in response to a database query. In other use cases, the reader 20 reconstructs and outputs a stream of records in sequence. Any other use cases can also apply.

[0028] In Figure 1 example, the reader 20 accesses a Parquet file 24 stored in dynamic random access memory (DRAM) 28. However, the reader 20 can typically access a Parquet file 24 stored on any other suitable memory or storage medium, or access a Parquet file 24 over a network, for example by communicating with a network interface controller (NIC).

[0029] Figure 1 The bottom inset shows the structure of the Parquet file 24. As described above, the file 24 includes a plurality of row groups; each row group includes a plurality of column blocks; and each column block includes a plurality of pages. Each page includes a header, followed by the repetition level for the page, followed by the definition level for the page, followed by the values for the page. The file 24 ends with a "footer" section that includes file metadata.

[0030] In this example, the reader 20 includes a memory interface 32 for accessing the Parquet file 24 in the memory 28. The reader 20 also includes a plurality of column reader circuits 36 (referred to for brevity as "column readers") and a record reconstruction circuit 40 (referred to for brevity as "record reconstructor"). Example implementations of the column readers 36 and record reconstructor 40 are described below in Figure 2 and Figure 4An overview of the Parquet reader 20 is depicted in

[0031] Generally, to reconstruct one or more records from the Parquet file 24, each column reader 36 is associated with a respective column. The column reader 36 then reads data from the respective column, optionally decrypts, decompresses and optionally decodes the data, and provides the decompressed data to the record reconstructor 40. The record reconstructor 40 reconstructs the records from the data provided by the column readers. Generally, the basic unit processed by each column reader 36 is a group of rows in a Parquet page.

[0032] Column reader and partial reader configuration Figure 2 is schematically shown according to an embodiment of the application Figure 1 A block diagram of a column reader 36 in the Parquet reader 20. The individual column readers 36 of the Parquet reader 20 generally all have a similar structure, and are assigned different column blocks during operation.

[0033] In the present example, the column reader 36 comprises a direct memory access (DMA) engine 44 for accessing the DRAM 28 and a plurality of partial readers 48 (repeat level reader 48A, definition level reader 48B and value reader 48C). The internal structure of the partial readers 48 according to an example embodiment is depicted in Figure 3

[0034] When the column reader 36 reads a particular Parquet page belonging to its assigned column block, the readers 48A, 48B and 48C read and decompress the repeat level, definition level and value parts of the page, respectively. The partial readers 48A and 48B are considered as examples of one or more structure information readers configured to read and decompress structure information from the column assigned to the column reader 36.

[0035] Generally, the compression mode used for compressing the repeat level, definition level and value parts (e.g. Gzip or Snappy) has a variable compression rate. Therefore, the size of the repeat level, definition level and value parts (after compression) is variable and data dependent. Consequently, the starting position of the definition level and value parts in the Parquet page is also variable and data dependent.

[0036] ​In various embodiments, the definition level reader 48B and the value reader 48C can use various techniques to determine their starting positions in the respective portions of a given Parquet page. In one embodiment, the definition level reader 48B starts reading and decompressing data from the beginning of the Parquet page (which begins with the page header and the repeat level portion), but discards the data until it reaches the start of the definition level portion. The purpose of this initial process is simply to identify the start of the definition level portion. Only from that point does the definition level reader 48B start outputting its decompressed data.

[0037] Similarly, the value reader 48C starts from the beginning of the Parquet page and reads and decompresses the data of the repeat level portion and the definition level portion in order to find the start of the value portion. The value reader 48C discards the decompressed data until it reaches the start of the value portion. Only from that point does the value reader 48C start outputting its decompressed data.

[0038] In one embodiment, upon starting to process a Parquet page, the readers 48B and 48C perform an initial process of finding the start of their respective portions. When all three portion readers 48A-48C are ready, they start reading and decompressing their respective portions in parallel in order to align (at least approximately) their output data. In this context, the term "align" means the association between the values and the structural information (e.g., repeat level and definition level) defined about the values.

[0039] In alternative embodiments, the column reader 36 need not include three separate portion readers 48 for reading the repeat level portion, the definition level portion, and the value portion. For example, the column reader 36 can include a single portion reader 48 coupled to three storage buffers. In such an implementation, the portion reader alternately reads and decompresses data from the repeat level portion, the definition level portion, and the value portion. The portion reader buffers the decompressed data of each portion in the respective buffer and generally maintains track of three pointers of the next position to read from each portion.

[0040] In yet another embodiment, the column reader can include a single portion reader 48 that: (i) reads and decompresses corresponding data from the repeat level portion and the definition level portion into a buffer, then (ii) reads and decompresses corresponding data from the value portion, and (iii) aligns the values with their repeat level and definition level.

[0041] Additionally, alternatively, the column reader 36 can be implemented in any other way and with any other suitable number of partial readers 48. In a typical implementation, the column reader reads, decompresses, and aligns one triplet of {repetition level, definition level, value} at a time, rather than reading and decompressing an entire Parquet page before aligning. In other words, the column reader typically reads, decompresses, and aligns one triplet before proceeding to read, decompress, and align the next triplet.

[0042] In Figure 2 Embodiments of the column reader 36 also include a microcontroller 60 configured to manage and control the operation of the column reader. Among other tasks, the microcontroller 60 is configured to parse the page header of the Parquet page being read. The microcontroller 60 stores the page header in a buffer 64.

[0043] In some embodiments, the column reader 36 also includes a cyclic redundancy check (CRC) block coupled to the DMA engine 44. The CRC block verifies the CRC on the repetition level data stream, the definition level data stream, and the value data stream. The CRC results are provided to the microcontroller 60.

[0044] The column reader 36 also includes control logic 52. Among other tasks, the logic 52 is configured to trigger the partial readers 48A, 48B, and 48C to provide the respective repetition level, definition level, and value, and to align the repetition level, definition level, and value provided by the partial readers. The logic 52 outputs the value to the record re- constructor 40 in parallel with the repetition level provided by the partial reader 48A.

[0045] In some embodiments, the logic 52 is configured to perform dictionary translation of at least some of the values before outputting the values to the record re- constructor 40. The dictionary or a portion thereof is cached in a dictionary cache 56 coupled to the logic 52. Various aspects of the dictionary translation will be discussed in further detail below.

[0046] Figure 3 is a block diagram of an example partial reader 48 in the column reader 36 according to embodiments of the application. Figure 2 The configuration can be used to implement the repetition level partial reader 48A, the definition level partial reader 48B, and / or the value partial reader 48C. The partial reader 48 is configured to read data from a portion of a Parquet page (repetition level portion, definition level portion, or value portion).

[0047] In this example, the partial reader 48 includes a configurable pipeline that includes (i) decryption logic 68 that decrypts data read from a partial, (ii) decompression logic 72 that decompresses the decrypted data, and (iii) a decoder 76 that decodes the decompressed data. A buffer 80 buffers the decrypted data between the decryption logic 68 and the decompression logic 72. A buffer 84 buffers the decompressed data between the decompression logic 72 and the decoder 76. Value processing logic 96 is configurable to perform a specified function on, or otherwise manipulate, the decoded values. Examples of functions are given in the "Operations on Read Values" section below. The value processing logic 96 outputs the decoded values (to Figure 2 the control logic 52 of the column reader 36).

[0048] The partial reader 48 also includes configuration registers 92 that hold available configurations for the decryption logic 68, the decompression logic 72, and the decoder 76. The configuration registers 92 of the decryption logic 68 can hold, for example, a decryption key to use to decrypt the partial. The configuration registers 92 of the decompression logic 72 can hold, for example, an identifier of a decompression mode to use and any related decompression parameters. The configuration registers 92 of the decoder 76 can hold, for example, an identifier of a decoding mode to use and any related decoding parameters. Any suitable decryption, decompression, and decoding modes supported by Parquet can be used.

[0049] The partial reader 48 also includes control logic 88 that configures and controls the various components of the partial reader.

[0050] One of the roles of the buffers 80 and 84 is to apply back pressure to the preceding pipeline stages. Aspects of the back pressure, and the use of back pressure for data alignment between different parts of a Parquet page and between different columns (or column blocks of row groups) of a Parquet file, are discussed further below.

[0051] In some embodiments, the partial reader 48 includes a dictionary buffer 100 coupled to the decoder 76. The buffer 100 can hold, for example, dictionary values used to store information about repetition levels and definition levels.

[0052] Record Reconstruction Figure 4 is schematically shown in accordance with an embodiment of the present invention Figure 1a block diagram of record reconstituter 40 in parquet reader 20. Reconstituter 40 receives data from the various column readers 36 and uses the data to reconstitute selected records of the Parquet file. Reconstituter 40 typically receives data from each column reader as three vectors— (i) a vector of values, (ii) a vector of repetition levels, and (iii) a vector of definition levels.

[0053] Reconstituter 40 includes a programmable record reader 104 that reconstitutes records from the data provided by the column readers. The reconstituted records are provided as output from the Parquet reader 20. Reconstituter 40 also includes a schema store 120 that stores the schema of the Parquet file being read.

[0054] In some embodiments, record reconstituter 40 can include a dictionary circuit 112 that performs dictionary translation of values and a dictionary cache 108 that stores dictionary entries. However, this is not mandatory— in some embodiments, all dictionary translation operations, including caching and cache lookups, are performed in the column readers 36. The dictionary translation feature, including various options for the "division of labor" between column readers and record reconstituter, will be set forth in further detail below.

[0055] In some embodiments, record reconstituter 40 also includes a function circuit 116 that performs a predefined function on the reconstituted records before outputting them. Some functions control the level to which record reconstituter 40 decomposes the nested structure of the records being reconstituted. Consider the example record described in the overview section above, which has two levels of nesting:

[0056] The record can also be represented as follows, still in fully nested format:

[0057] Decomposing one level of nesting will yield three records:

[0058]

[0059]

[0060] Decomposing two levels of nesting will yield six flat (non-nested) records:

[0061]

[0062]

[0063]

[0064]

[0065]

[0066] In some embodiments, the function circuit 116 is configured to specify the number of levels to be decomposed in reconstructing a record, e.g., per record or as a general mode of operation. The number ranges from zero to the maximum nesting level.

[0067] In some embodiments, the programmable record reader 104 in the reconstructor 40 includes multiple processing engines (e.g., cores or processors) configured to reconstruct multiple respective records simultaneously. The reader 104 can include a scheduler that flexibly assigns records to the processing engines.

[0068] In some embodiments, the record reconstructor 40 is configured to reconstruct multiple streams of records in parallel or alternately (e.g., using different processing engines simultaneously). Each stream of records typically originates from a different query and thus typically has separate definitions of columns to read, functions to apply, etc. In some embodiments, the record reader 104 in the reconstructor 40 is configured to apply backpressure to the column reader 36 only for one or more selected streams of records, and to avoid using backpressure for other streams.

[0069] When reconstructing multiple streams of records alternately, the reader 104 can switch between record streams once all the decompression buffers 84 are empty (at the end of each group of rows). Alternatively, the reader 104 can switch between record streams while some decompression buffers are non-empty, either by saving the current state of the decompression buffers ("context") or by repeating decompression when the same record stream is rescheduled for reconstruction. In some embodiments, the record reader 104 in the reconstructor 40 is configured to use context switching only for one or more selected streams of records, and to wait until the end of a group of rows for other streams. In any case, pausing or resuming a certain stream does not affect other streams. Figure 3

[0070] ​In various embodiments, the readers 20 can apply cooperative context switching between record streams, or preemptive context switching between record streams. In cooperative context switching, an external controller allows the readers 20 to complete reconstruction of an entire group of rows, and then only configures the readers to reconstruct a group of rows for a different context (typically a different query related to a different set of columns). In preemptive context switching, if a reader 20 does not complete reconstruction of a group of rows after a predefined period of time, the controller instructs the reader 20 to stop in the middle of the group of rows. The controller saves the context specifying the reader's state in the middle of the group of rows (e.g., the page index of each column reader and the value index in each page) and configures the reader to operate on another group of rows. The controller can later configure the reader to resume reading the preempted group of rows from the point at which it was preempted.

[0071] In various embodiments, the record reconstructor 40 can output reconstructed records to any suitable "user system," e.g., a processor that applies further processing to the records. In some embodiments, the reconstructor 40 is able to receive and act on backpressure signaling (e.g., signals or notifications) from the user system. Typically, the user system sends a "pause" notification when it is unable to receive new records, and a "resume" notification when it is again able to receive records. Upon receiving a "pause" notification, the record reconstructor 40 can start buffering reconstructed records as space permits. Otherwise, the record reconstructor can propagate backpressure forward to the column readers.

[0072] In some embodiments, the record reconstructor 40 outputs various types of metadata along with the reconstructed records. The metadata can include, for example, additional information related to particular fields, such as a Boolean value indicating whether a field value satisfies a condition, a numeric value indicating the first letter of a string field, etc. As another example, the metadata can indicate nesting levels (or other structural information) in a manner different from that specified in the Parquet format. For example, the metadata can indicate whether a nested value in a field is null.

[0073] Additional embodiments and variations Using reverse pressure alignment In some embodiments, the record reconstructor 40 and the column readers 36 (including the partial readers 48) use a backpressure mechanism to control the rate at which data is read from the various partials of various column blocks. For example, this rate control is important to maintain alignment between data read from different column blocks.

[0074] In example implementations, each column reader 36 operates independently of the others under control of a record reader 104 of the record reconstitutor 40. The record reader 104 allocates respective column chunks to each column reader and instructs the column reader which Parquet pages to read from the column chunk.

[0075] As each column reader 36 operates independently, and as data size and compression rates are variable, some column readers can return data at a higher rate than others. However, the record reader 104 aims to maintain alignment between the column readers 36, i.e. to ensure that the data provided by different column readers at a given time corresponds to the same record or a small number of adjacent records.

[0076] In example embodiments, the record reader 104 can detect that, at a given time, the data provided by the various column readers belongs to more than a predefined maximum number of adjacent records. In response to detecting this misalignment, the record reader 104 can temporarily pause one or more of the relatively faster column readers in order to allow the slower column reader(s) to catch up. When alignment is regained, i.e. when the data provided by the various column readers belongs to no more than the predefined maximum number of adjacent records, the record reader 104 can resume the paused column readers.

[0077] A typical example where alignment is required is when one column (denoted as col 1) stores an integer field, while another column (denoted as col 2) holds a corresponding list. In this case, for each record, the column reader assigned to col 1 needs to read a single value, while the column reader assigned to col 2 needs to read the entire list.

[0078] Typically, each column reader 36 internally passes backpressure to the respective part readers 48 and within the pipeline of each part reader 48. For example, when a particular column reader 36 is paused by the record reader 104, the decoder 76 of the part reader 48 will be paused, and when resumed, will be resumed. Within a given part reader 48 (see Figure 3 ), when the decoder 76 is paused, the buffer 84 will gradually fill with decompressed data. When a particular fill level is exceeded, the buffer 84 will pause the decompression logic 72. Pausing the decompression logic 72 will cause the buffer 80 to gradually fill with decrypted data. When a particular fill level is exceeded, the buffer 80 will pause the decryption logic 68.

[0079] When the column reader resumes operation, the resumption is internally propagated to the partial readers in a similar fashion - each buffer (84 and 80) begins to empty, and when sufficient empty, resumes the previous pipeline stage. In various embodiments, the record reader 104 and the column reader 36 (including the partial readers 48 and their internal buffers and pipeline stages) can use any suitable signaling to apply back pressure. For example, the "ready" signals between stages can be de-asserted to pause, and de-asserted to resume operation.

[0080] The record reconstitutor 40 generally uses available information about the document being read, such as structural information and / or schema of the document, to maintain alignment between values provided by the various column readers 36. In some embodiments, the record reconstitutor determines the size of data that each record needs from each column reader, and maintains alignment between column readers by obtaining the appropriate size of data from each column reader.

[0081] For example, if none of the columns being read have any nesting, the record reconstitutor 40 simply takes the value from each column. If a certain field is wider (wider than the size of the bus), the record reconstitutor can need to receive that field from the column reader in multiple cycles per record, while only one cycle is needed for every other field. For variable size fields, such as strings, the record reconstitutor can check the length of the field, which is typically specified in the first few bytes of the field. The record reconstitutor can then use the length information to set the number of cycles needed to receive the field from the column reader. When a column has nesting, the structural information (e.g., the repeat level and definition level in Parquet) helps the record reconstitutor decide how to align the column with other columns and reconstitute the record.

[0082] In some cases, the number of columns requested to be read by the Parquet reader 20 is greater than the number of column readers 36 in order to reconstitute a record. In some embodiments, the record reconstitutor 40 handles such requests by reconstituting partial records based on a partial subset of the columns that does not exceed the number of column readers, and saving the partial records to memory. The set of partial records can then be merged (e.g., by an external controller or user system) to form a fully reconstituted record with any desired number of columns.

[0083] Dictionary translation According to the Parquet format, a given column block may optionally contain a dictionary. The dictionary represents common values ​​(e.g., strings) with shorter keys. Subsequent Parquet pages within the column block include the short keys instead of the longer actual values, which further reduces the file size. Dictionaries can be used to represent strings and / or any other supported value types, such as double or long. Dictionaries can represent values ​​with a constant size (i.e., all values ​​in the dictionary have the same size) or values ​​with a variable size (i.e., different values ​​in the dictionary can differ in size from each other). Dictionaries are typically placed at the beginning of the column block, before the first Parquet page, and are often compressed using, for example, Gzip or Snappy.

[0084] In some embodiments, the Parquet reader 20 is configured to read, decompress, and store one or more dictionaries from one or more column blocks to translate the keys read from the column blocks into actual values ​​on the fly, and to reconstruct and output records with actual values.

[0085] In this embodiment, when a dictionary is encountered, reader 20 uses one of the partial readers 48 to decompress the dictionary and stores the decompressed dictionary in a nearby memory, such as static random access memory (SRAM). When the key is later translated into an actual value, the circuitry in reader 20 uses the key to derive the memory address where the corresponding actual value is stored.

[0086] Figure 5 This is a schematic illustration based on an embodiment of the present invention. Figure 1 A block diagram of the dictionary translation circuit 122 in the Parquet reader 20. Circuit 122 can be implemented, for example, as logic 52 and dictionary cache 56 in the column reader 36. Figure 2 Part of the dictionary circuit 112 in the record reconstructor 40 () Figure 4 It can be part of, or implemented in, both of the dictionary. As another example, under the assumption that not all column readers will access the dictionary simultaneously, appropriate memory can be used to share the dictionary's storage among two or more column readers. In yet another example, the dictionary is stored in a memory subsystem, and the relevant portion of the memory is cached in a nearby close-by cache memory.

[0087] Dictionary circuit 122 receives keys read from a Parquet page as input and outputs values ​​represented by the keys according to the dictionary. Circuit 122 includes an indirect addressing table 124 and a map 128. Map 128 stores the actual values ​​and is accessed via addresses (represented as offsets from some base address). Table 124 and map 128 are referred to herein as "dictionary data structures". In alternative embodiments, any other suitable type of dictionary data structure may be used.

[0088] For a dictionary of constant-size values, circuit 122 directly calculates the offset based on the key, for example, by multiplying the key by the constant size of the value. This offset is then used to access map 128 and extract the value from it. In this example, indirect addressing table 124 is not used. For a dictionary of variable-size values, circuit 122 uses the key to access indirect addressing table 124 and extracts the offset and size from it. The offset and size are then used to access map 128 and extract the value from it. In an alternative embodiment, indirect addressing table 124 can also be used to access a dictionary of constant-size values ​​without using multiplication.

[0089] Typically, circuit 122 constructs a dictionary (e.g., filling mapping 128 and indirect addressing table 124, if used) when reading a dictionary from a column block. Later, when reading a Parquet page from a column block, circuit 122 performs an on-the-fly translation using the filled mapping 128 and the filled table 124, if used.

[0090] Operations on read values In some embodiments, the Parquet reader 20 is configured to modify values ​​read from a Parquet file and insert the modified values ​​into the reconstructed record. Typically, an operation is specified for a selected field (selected column). This operation can be performed at any stage of the read and reconstruct process, such as before record reconstruction, or after record reconstruction and before outputting the reconstructed record.

[0091] For example, it can be done in column reader 36 (e.g., via...) Figure 3 Value processing logic 96 in partial reader 48 or in record reconstructor 40 (e.g., via...) Figure 4 The function circuit 116) performs value operations. Operations performed in the column reader are typically (though not necessarily) simple, hardware-controlled operations, such as comparisons of values ​​with thresholds or simple logic functions. Operations performed in the record reconstructor can be more complex, software-controlled operations, such as operations derived from queries that trigger read operations.

[0092] Reader 20 can perform any suitable type of operation on the value. Several illustrative, non-restrictive examples of operations include the following (regarding the field (column) denoted as "field1"): • Apply a function to the value of a field, e.g., Round(fieldl), fieldl*4, Sign(fieldl), etc.

[0093] Output a Boolean flag ("0" or "1") instead of fieldl, depending on whether the condition defined on fieldl is true or false, e.g., "fieldl>6". Another example of a Boolean function is the following function: return "1" if the specified value exists in the list, otherwise return "0".

[0094] Operate on nested values, e.g., modify a list or selected items on the entire list, or apply a function to a list, e.g., find the maximum, minimum, or sum of values in the list.

[0095] Perform multiple operations on a single field, e.g., output multiple Boolean flags in response to multiple conditions defined on the same field (e.g., "fieldl>7", "fieldl<30", "fieldl>100").

[0096] • Perform various functions on strings, e.g., logical functions like col1=="the" or col1=="%the", where % is a wildcard.

[0097] Perform various functions on date types, e.g., a function to extract the year and month from a date field in col4.

[0098] • Filter (i.e., selectively discard or retain) records based on criteria defined on one or more field values, e.g., by fusing operations on multiple fields. Filtering can be performed before or after records are reconstructed.

[0099] In various embodiments, the reader 20 can perform various filtering operations to decide which records to output and which records to discard. In some embodiments, the filter is based on a single column, e.g., col 1>7, in which case the record reconstructor 40 only outputs records that meet the specified condition and discards records that do not meet the specified condition. Note that the output is a complete record (e.g., col 1, col 2,..., col 8), even though the condition pertains to only a particular column.

[0100] In another embodiment, the filter is based on a combination of two or more columns, e.g., "(Col 1>7 and col 2=="The record") or (extract_date(col 3)-current_date<50)". In this embodiment, the columns in the filter definition can be of different types. Furthermore, various operations can be performed on the columns in order to arrive at the final decision regarding whether to output or discard a particular record.

[0101] In some cases, the filter does not necessarily relate to the records output from the rewriter 40. Additionally or alternatively, the records being output can be different from the original records in the file. For example, consider the filter "(Col 1 > 17 and col2 in ("The record", "The sum", "The Best")) or (extract_date(col3) - current_date < 10)". In one embodiment, the record rewriter 40 checks whether the records satisfy the condition. Upon identifying records that satisfy the condition, the record rewriter 40 outputs corresponding records with respect to different columns, e.g., "col 1 -5, col2, (if col2 == "The Best", 1, 0), extract_date(col3), extract_month(col3), col4, col5, col6, 7*col7-43, is_Null(col8)".

[0102] The above filters are given purely by way of example, in order to demonstrate the capabilities provided by filtering in the reader 20. In alternative embodiments, any other suitable filters can be used.

[0103] In general, when a value after an operation is derived from one or more fields of a record (e.g., when a function is performed on one or more fields), the reader 20 outputs the value after the operation in the metadata accompanying the restructured record. In some embodiments, the reader 20 outputs the value after the operation, rather than the field value read from the column. In other embodiments, the reader 20 outputs both the value after the operation and the value read from the column. This can generally be done by outputting an additional field.

[0104] In some embodiments, although the reader 20 is generally configured based on the file being read, the record rewriter 40 can configure value modifications based on the processing (e.g., the query being processed) rather than the file, and thus further optimize the processing.

[0105] Overall method description Figure 6 is a flowchart schematically illustrating a method for reading out from a Parquet file for hardware implementation, performed by the Parquet reader 20, according to an embodiment of the present application. The method begins at a column specification step 130, in which the record rewriter 40 specifies a set of columns to be read from the Parquet file, e.g., in response to a query. At a configuration step 134, the record rewriter 40 assigns a respective column reader 36 to each column to be read, and configures the column readers 36 accordingly.

[0106] The following steps are performed by the column reader 36 assigned to the column on each Parquet page of each column chunk being read (138-146). In a partial initialization step 138, the column reader 36 initializes the partial reader 48 with the starting locations of the repetition level, definition level, and value portions to be read from in the partial initialization step 138. In a readout step 142, the partial reader 48 reads, decrypts, decompresses, and decodes the repetition level, definition level, and values from the Parquet page. The column reader 36 outputs the repetition level, definition level, and values to the record reconstructor 40.

[0107] In a record reconstruction step 150, the record reconstructor 150 reconstructs one or more records from the repetition level, definition level, and values provided by the respective column reader 36.

[0108] The configuration of the Parquet reader 20 and its components (e.g., the column readers 36, the partial readers 48, and the record reconstructor 40) as shown is an example configuration depicted purely for conceptual clarity. In alternative embodiments, any other suitable configuration can be used. Figures 1-5

[0109] The Parquet reader 20 and its components (e.g., the column readers 36, the partial readers 48, and the record reconstructor 40) can be implemented using any suitable hardware (e.g., in an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA)).

[0110] In some embodiments, particular portions of the Parquet reader 20 and its components (e.g., the column readers 36, the partial readers 48, and the record reconstructor 40) can be implemented using a programmable processor programmed in software to perform the functions described herein. The software can be downloaded to the processor in electronic form, over a network, for example; or alternatively or additionally the software can be provided and / or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.

[0111] It is to be understood that the embodiments which have been described above are cited by way of example, and that the present application is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present application includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons of ordinary skill in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in such incorporated documents in a manner that conflicts with the definitions made explicit herein, the definitions in the present specification are intended to prevail.​

Claims

1. A hardware-implemented file reader, comprising: an interface configured to access a file comprising a plurality of records, wherein the records store values according to a structure that supports optional values and repeating values, and wherein the file is stored in a column format having a plurality of columns, each column storing (i) compressed values and (ii) corresponding compressed structure information associating the values in the column with the nested structure of the records; a plurality of hardware-implemented column readers, each column reader configured to be assigned to a respective selected column and to read and decompress the values and the structure information from at least a portion of the selected column, wherein at least a given one of the column readers comprises a hardware-implemented pipeline configured to incrementally decrypt, decompress, and decode portions of the values or the structure information; and a hardware-implemented record reconstructor configured to reconstruct one or more of the records from at least a portion of the columns read by the column readers and to output the reconstructed records.

2. The file reader of claim 1, wherein, the hardware-implemented pipeline comprises: decryption logic configured to decrypt the values or the structure information; a first buffer configured to buffer decrypted values or structure information produced by the decryption logic; decompression logic configured to decompress the decrypted values or structure information buffered in the first buffer; a second buffer configured to buffer decompressed values or structure information produced by the decompression logic; and a decoder configured to decode the decompressed values or structure information buffered in the second buffer.

3. The file reader of claim 2, wherein, the given column reader, which includes at least the hardware-implemented pipeline, is implemented within an integrated circuit (IC), the hardware-implemented pipeline including the first buffer and the second buffer.

4. The file reader of claim 2, wherein, the second buffer is configured to apply back pressure to the decompression logic, and the first buffer is configured to apply back pressure to the decryption logic, thereby limiting a rate at which the selected column is read out.

5. The file reader of claim 1, wherein, the given column reader or the record reconstructor is configured to operate on at least some of the decoded values or structure information.

6. The file viewer as recited in claim 1, wherein, the given column reader or the record reconstructor includes a hardware-implemented filter configured to filter the records based on one or both of (i) criteria defined on one or more of the values and (ii) a received query.

7. The file reader of claim 1, wherein, the given column reader is configured to align at least some of the decompressed values with corresponding decompressed structure information before reading and decompressing subsequent values and subsequent structure information from the selected column.

8. A method for hardware-implemented file readout, comprising: accessing a file using a plurality of hardware-implemented column readers, wherein the file comprises a plurality of records, wherein the records store values according to a structure that supports optional values and repeating values, and wherein the file is stored in a column format having a plurality of columns, each column storing (i) compressed values and (ii) corresponding compressed structure information associating the values in the column with the nested structure of the records; Assigning each column reader to a respective selected column, and reading and decompressing the values and the structure information from at least a portion of the selected column, includes, in at least a given one of the column readers, incrementally decrypting, decompressing, and decoding portions of the values or the structure information using a hardware-implemented pipeline; and Reconstructing, using a hardware-implemented record reconstructor, one or more of the records from at least a portion of the columns read by the column readers, and outputting the reconstructed records.

9. The method of claim 8, wherein, Incrementally decrypting, decompressing, and decoding the portions of the values or the structure information includes: Decrypting the values or the structure information using decryption logic; Buffering the decrypted values or structure information produced by the decryption logic in a first buffer; Decompressing the decrypted values or structure information buffered in the first buffer using decompression logic; Buffering the decompressed values or structure information produced by the decompression logic in a second buffer; and Decoding the decompressed values or structure information buffered in the second buffer using a decoder.

10. The method of claim 9, wherein, The given column reader, including at least the hardware-implemented pipeline, is implemented within an integrated circuit (IC), the hardware-implemented pipeline including the first buffer and the second buffer.

11. The method of claim 9, further comprising: Applying back pressure to the decompression logic using the second buffer, and applying back pressure to the decryption logic using the first buffer, thereby limiting a rate at which the selected column is read out.

12. The method of claim 8, further comprising: Operating on at least some of the decoded values or structure information in the given column reader or the record reconstructor.

13. The method of claim 8, further comprising: In the given column reader or the record reconstructor, using a hardware-implemented filter to filter the records based on one or both of (i) criteria defined for one or more of the values, and (ii) a received query.

14. The method of claim 8, further comprising: Aligning at least some of the decompressed values with corresponding decompressed structure information prior to reading and decompressing subsequent values and subsequent structure information from the selected column.

Images