A decompression implementation method and device based on a DPU acceleration card under a heterogeneous architecture

Abstract

The application provides a decompression implementation method and device based on a DPU acceleration card under a heterogeneous architecture, the method comprising: the DPU acceleration card receiving compressed data streams offloaded by a CPU server through an established connection and storing the compressed data streams on a DDR storage unit of the DPU acceleration card; sequentially reading sub-data streams of a preset byte length in the compressed data streams from the DDR storage unit into an acceleration core, and performing decompression processing on the sub-data streams by the acceleration core, comprising: comparing and finding the starting position of a compressed file in the sub-data streams by using a sliding window mode; for compressed blocks constituting the compressed file, decompressing data contained in each compressed block in the compressed data streams is obtained according to a preset decompression mode; the decompressed data contained in each compressed block is merged into one original data; after decompression by the acceleration core, the original data obtained after decompression is written back to the DDR storage unit, and the original data is transmitted back to the CPU server through the established connection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of decoding implementation under a heterogeneous architecture, and in particular to a decompression implementation method and device based on a DPU acceleration card under a heterogeneous architecture. BACKGROUND

[0002] With the development of technology, the continuous improvement of disk read and write and network bandwidth performance, the performance bottleneck in the database and big data fields has shifted to computing power, and the existing CPU scheme cannot provide sufficient computing power to support the existing big data business scale.

[0003] The existing decompression algorithm, such as snappy decompression, has the following disadvantages: (1) the snappy decompression function can only be applied to the CPU scene, relying on the CPU server to call the decompression function. That is, the snappy decompression work can only be implemented on the CPU server; (2) the CPU instruction set architecture cannot efficiently and quickly decompress a large amount of snappy compressed data; (3) the CPU decompression process will generate a high CPU occupancy rate, making the server unable to perform other tasks.

[0004] Therefore, how to build a more powerful decompression implementation scheme with a heterogeneous system is a technical problem to be solved. SUMMARY

[0005] In view of this, the embodiments of the present application provide a decompression implementation method based on a DPU acceleration card under a heterogeneous architecture to eliminate or improve one or more defects in the prior art.

[0006] One aspect of the present application provides a decompression implementation method based on a DPU acceleration card under a heterogeneous architecture, the DPU acceleration card and the CPU server establish a connection to form a heterogeneous system, the DPU acceleration card includes a DDR storage unit and an acceleration core, and the method comprises the following steps:

[0007] The DPU acceleration card receives the compressed data stream offloaded by the CPU server through the established connection, and stores the compressed data stream on the DDR storage unit of the DPU acceleration card; wherein the compressed data stream includes a compressed file, the compressed file is composed of a plurality of compressed blocks, and the compressed blocks carry block type information;

[0008] Sub-data streams of a preset byte length in the compressed data stream are sequentially read from the DDR storage unit into the acceleration core, and the acceleration core performs decompression processing on the sub-data streams;

[0009] The step of accelerating the decompression of the sub-data stream includes: using a sliding window to search for the start position of the compressed file in the sub-data stream, and the start position of the compressed file is identified by block type information; for a compressed block constituting a compressed file, decompressed data contained in each compressed block in the compressed data stream is obtained by using a preset decompression mode; the decompressed data contained in each compressed block is merged into one original data; and the original data obtained after decompression is written back to the DDR storage unit through the established connection to the CPU server.

[0010] In some embodiments of the present application, each compressed block further contains block length information and block data content, and the step of using a sliding window to search for the start position of the compressed file in the sub-data stream includes: using a sliding window to search for the position of the first compressed block of the compressed file in the sub-data stream, thereby determining the start position of each compressed file.

[0011] In some embodiments of the present application, the step of obtaining the decompressed data contained in each compressed block in the compressed data stream by using a preset decompression mode includes: using a sliding window to parse the block data content from the compressed block according to the block length information, the block data content contains length information and compressed data, and the compressed data meeting the length information is read from the block data content; and the decompressed data contained in each compressed block is decompressed from the compressed data by using the tag contained in the compressed data.

[0012] In some embodiments of the present application, the step of merging the decompressed data contained in each compressed block into one original data includes: using a multiplexer to merge the decompressed data contained in each compressed block obtained based on different tags into one original data.

[0013] In some embodiments of the present application, the block data content further contains a cyclic redundancy check bit used for detecting or checking decoding data errors; and the length information contained in the block data content is encoded by using the varint encoding rule.

[0014] In some embodiments of the present application, the type of the tag includes a literal tag and a copy tag; and the step of decompressing the decompressed data contained in each compressed block from the compressed data by using the tag contained in the compressed data includes: when the type of the tag is the literal tag, directly taking the data of the specified length in the compressed data as the decompressed data; and when the type of the tag is the copy tag, copying the data of the specified length from the previous data according to the offset position contained in the compressed data as the decompressed data.

[0015] In some embodiments of the present application, the replication tag is further divided into multiple replication forms according to length information and offset position, and the compression data is identified by a replication form keyword; the previous data is cached by a pre-designed RAM memory, a read address register and a write address register on the DPU acceleration card for reading and replication.

[0016] In some embodiments of the present application, the preset decompression mode includes any one of snappy decompression, Gzip decompression, LZ4 decompression and Brotli decompression.

[0017] Another aspect of the present application provides a decompression implementation device based on a DPU acceleration card under a heterogeneous architecture, the DPU acceleration card and a CPU server establish a connection to form a heterogeneous system, the DPU acceleration card includes a DDR storage unit and an acceleration core, and the device includes: the DDR storage unit, used for receiving and storing a compressed data stream offloaded by the CPU server through the established connection, reading a sub-data stream of a preset byte length in the compressed data stream from the DDR storage unit to the acceleration core in sequence, and performing decompression processing on the sub-data stream by the acceleration core; wherein the compressed data stream includes a compressed file, the compressed file is composed of multiple compressed blocks, and the compressed blocks carry block type information; the acceleration core, used for performing decompression processing on the sub-data stream, specifically including: comparing and finding the start position of the compressed file in the sub-data stream by using a sliding window, and the start position of the compressed file is identified by the block type information; for the compressed blocks constituting the compressed file, the decompression data contained in each compressed block in the compressed data stream is obtained by parsing according to a preset decompression mode; the decompression data contained in each compressed block obtained by decompression is merged into one way of original data; the DDR storage unit is further used for writing back the original data obtained after decompression by the acceleration core, and transmitting the original data back to the CPU server through the established connection.

[0018] In some embodiments of the present application, the sliding window register is pre-configured in the decompression implementation device based on the DPU acceleration card under the heterogeneous architecture for the acceleration core to perform decompression processing on the sub-data stream, the sliding window register includes: a first sliding window register, configured to find the position of a first compressed block of a compressed file in the sub-data stream in a sliding window manner, so as to determine the starting position of each compressed file; wherein each compressed block further contains block length information and block data content; a second sliding window register, configured to parse the block data content from the compressed block according to the block length information in a sliding window manner, the block data content contains length information and compressed data, and the compressed data meeting the length information is read from the block data content; decompression data contained in each compressed block is obtained from the compressed data based on a label contained in the compressed data; and a third sliding window register, configured to merge the decompression data contained in each compressed block obtained based on different labels into one piece of original data in a sliding window manner and by using a multiplexing switch.

[0019] The decompression implementation method and device based on the DPU acceleration card under the heterogeneous architecture can offload the compressed data stream to be decompressed to the DPU acceleration card, gradually parse the decompression data contained in the compressed data stream by using the DPU acceleration card in a sliding window manner, and integrate the decompression data to obtain one piece of original data as output, so as to realize decompression from the compressed data stream to the original data under the heterogeneous system.

[0020] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following detailed description and drawings in which

[0021] Those skilled in the art will appreciate that the objects and advantages of the application can not be limited to the specifically described above, and the above and other objects of the application can be more clearly understood according to the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0022] The drawings described herein are used to provide further understanding of the application, constitute a part of the present application, and do not constitute a limitation of the application. In the drawings:

[0023] Figure 1 The flowchart of the decompression implementation method based on the DPU acceleration card under the heterogeneous architecture in an embodiment of the present application.

[0024] Figure 2 The schematic diagram of the heterogeneous architecture.

[0025] Figure 3Figure 1 is a flowchart illustrating the implementation steps of decompression based on a heterogeneous architecture DPU accelerator card according to an embodiment of the present application.

[0026] Figure 4 Figure 2 is a schematic diagram of the external interface of an acceleration core according to an embodiment of the present application.

[0027] Figure 5 Figure 3 is a flowchart illustrating the processing procedure of an acceleration core according to an embodiment of the present application.

[0028] Figure 6 Figure 4 is a schematic diagram of the compressed block format included in a compressed file according to an embodiment of the present application.

[0029] Figure 7 Figure 5 is a schematic diagram of the pointer indicating the write position of a sliding window according to an embodiment of the present application.

[0030] Figure 8 Figure 6 is a schematic diagram of the extraction and output procedure of data content in a compressed block according to an embodiment of the present application.

[0031] Figure 9 Figure 7 is a schematic diagram of the structure of data content included in a compressed block according to an embodiment of the present application.

[0032] Figure 10 Figure 8 is a flowchart illustrating the procedure of parsing the length information of a varint encoding according to an embodiment of the present application.

[0033] Figure 11 Figure 9 is a schematic diagram of the processing sequence of parsing the length information of a varint encoding according to an embodiment of the present application.

[0034] Figure 12 Figure 10 is a schematic diagram of the procedure of parsing a varint encoding by splicing according to an embodiment of the present application.

[0035] Figure 13 Figure 11 is a schematic diagram of the structure of compressed blocks of different labels according to an embodiment of the present application.

[0036] Figure 14 Figure 12 is a schematic diagram of the write address register and the read address register according to an embodiment of the present application.

[0037] Figure 15 Figure 13 is a schematic diagram of the principle of reading the RAM register according to an embodiment of the present application.

[0038] Figure 16 Figure 14 is a schematic diagram of synthesizing one way of original data from literal label data and copy label data according to an embodiment of the present application.

[0039] Figure 17 Figure 15 is a schematic diagram of the principle of merging multiple ways of data by using a sliding window according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and drawings. Herein, the illustrative embodiments of the present application and their descriptions are used to explain the present application but not as a limitation of the present application.

[0041] It should be noted here that, in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0042] It should be emphasized that the term "comprising / comprising" as used herein refers to the presence of a feature, element, step or component, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0043] It should be noted here that, if not specifically stated, the term "connection" as used herein can not only mean direct connection, but also indirect connection with the presence of an intermediate.

[0044] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts or the same or similar steps.

[0045] In order to overcome the problems existing in the prior art, the present application aims to propose a new decompression implementation method and device for heterogeneous architecture in the field of database and big data, which relies on a heterogeneous computing system composed of opinion DPU accelerator and traditional CPU server to realize the replacement of existing pure CPU server, while greatly improving the decompression efficiency.

[0046] Figure 1 The flow chart of the decompression implementation method based on DPU accelerator under heterogeneous architecture in an embodiment of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the DPU accelerator and the CPU server establish a connection to form a heterogeneous system, and the DPU accelerator includes a DDR storage unit and an acceleration core. The decompression implementation method based on DPU accelerator under heterogeneous architecture proposed by the present application includes the following steps:

[0047] Step S100: The DPU accelerator receives the compressed data stream offloaded by the CPU server through the established connection, and stores the compressed data stream on the DDR storage unit of the DPU accelerator; wherein the compressed data stream includes a compressed file, and the compressed file is composed of a plurality of compressed blocks, and the compressed blocks carry block type information.

[0048] Step S200: sequentially read the sub-data stream of a preset byte length in the compressed data stream from the DDR storage unit into the acceleration core, and perform decompression processing on the sub-data stream by the acceleration core.

[0049] Step S300: write the original data obtained after decompression back to the DDR storage unit after acceleration core decompression, and the original data is transmitted back to the CPU server through the established connection, realizing decompression from compressed data stream to original data in a heterogeneous system.

[0050] In the step S200, the step of decompressing the sub-data stream by the acceleration core specifically includes:

[0051] Step S210: the step of decompressing the sub-data stream by the acceleration core includes: using a sliding window to compare and find the starting position of the compressed file in the sub-data stream, and the starting position of the compressed file is identified by block type information.

[0052] In the specific implementation process, each compressed block further includes block length information and block data content, and the step of using a sliding window to compare and find the starting position of the compressed file in the sub-data stream in step S210 includes: using a sliding window to compare and find the position of the first compressed block of the compressed file in the sub-data stream, so as to determine the starting position of each compressed file.

[0053] By using the embodiment of the application, the position of the compressed file contained in the compressed data stream can be determined through the preset sliding window register on the DPU acceleration card, so that the compressed file is parsed based on the preset decompression rule, and the decompression from the compressed data stream to the original data in the heterogeneous system is realized.

[0054] Step S220: for the compressed blocks constituting the compressed file, the decompressed data contained in each compressed block in the compressed data stream is parsed according to the preset decompression mode.

[0055] In the specific implementation process, the step of parsing the decompressed data contained in each compressed block in the compressed data stream according to the preset decompression mode in step S220 includes: using a sliding window to parse the block data content from the compressed block according to the block length information, the block data content includes length information and compressed data, and the compressed data conforming to the length information is read from the block data content; and using the label contained in the compressed data to decompress the decompressed data contained in a single compressed block from the compressed data.

[0056] By using the embodiment of the application, each compressed block contained in the compressed file can be decompressed through the sliding window register, so that the block data content contained in the compressed block is obtained, and the compressed data is parsed based on the block data content. In a popular sense, the compressed data is part of the original data, and the decompressed data obtained based on each compressed block needs to be spliced to obtain the original data.

[0057] Step S230: merging the decompressed data contained in each compressed block into one original data.

[0058] In the implementation, the step of merging the decompressed data contained in each compressed block into one original data in step S230 includes: merging the decompressed data contained in each compressed block based on different tags into one original data by using a multiplexer.

[0059] In summary, the decompression implementation method based on the DPU acceleration card under the heterogeneous architecture can offload the compressed data stream to be decompressed to the DPU acceleration card, use the DPU acceleration card to gradually parse the decompressed data contained in the compressed data stream in a sliding window manner, and integrate one original data as output to realize the decompression from the compressed data stream to the original data under the heterogeneous system.

[0060] In some embodiments of the present application, the block data content further contains a cyclic redundancy check bit for detecting or checking decoding data errors. The length information contained in the block data content is encoded using the varint encoding rule.

[0061] Using the embodiment of the present application, using varint encoding can save storage space and improve space utilization, and without specifying the length in numerical value, it can adapt to different sizes of integers, and can more efficiently realize compression and decompression.

[0062] In some embodiments of the present application, the types of tags include a literal tag and a copy tag. The step of decompressing the decompressed data contained in a single compressed block from the compressed data using the tags contained in the compressed data in step S230 includes: when the tag type is a literal tag, directly taking the data of the specified length in the compressed data as the decompressed data; and when the tag type is a copy tag, copying the data of the specified length from the previous data according to the offset position contained in the compressed data as the decompressed data.

[0063] Using the embodiment of the present application, the compressed block using the literal tag can store non-repeated information, and the compressed block using the copy tag can store information that exists repeatedly.

[0064] In some embodiments of the present application, the copy tag is further divided into multiple copy forms according to the length information and the offset position, and the compressed data uses a copy form keyword to identify. And the previous data is cached by a pre-designed RAM memory, a read address register and a write address register on the DPU acceleration card for reading and copying.

[0065] In some embodiments of the present application, the selected decompression mode includes any one of snappy decompression, Gzip decompression, LZ4 decompression and Brotli decompression. In the present application, snappy decompression is mainly used as an example. Snappy is an open source data compression library, and its main feature is the high speed of compression and decompression, rather than pursuing the ultimate compression rate. Snappy is a fast compression and decompression algorithm developed by Google, which aims to provide high speed of compression and decompression while maintaining a relatively high compression ratio, although this compression ratio may be relatively lower than some other algorithms.

[0066] Figure 2 The schematic diagram of the heterogeneous architecture. The CPU server and the DPU acceleration card are connected through the PCIe channel. The CPU server includes a CPU and a memory, and the DPU acceleration card includes an interface for connection with the outside world, a DDR and an acceleration core.

[0067] The overall scheme of the present application is as follows: (1) The DPU acceleration card has a standard PCIe interface, which is inserted into the CPU server through the PCIe slot, and data is transmitted through the PCIe. (2) Optionally, the DPU acceleration card provides a fiber interface or a network port or other data transmission interface. (3) The DPU acceleration card includes a DDR storage unit and an acceleration core for operation. (4) The compressed data is transmitted from the CPU memory to the DDR of the DPU acceleration card through the PCIe; in addition, the compressed data can also be input from the outside through the fiber interface or the network port or other interfaces and stored in the DDR of the DPU acceleration card. (5) The acceleration core reads the compressed data from the DDR of the DPU acceleration card. (6) After the decompression of the acceleration core, the output decompressed data is written back to the DDR of the DPU acceleration card. (7) Finally, the data is transmitted back to the CPU memory through the PCIe, completing the whole process of heterogeneous acceleration of snappy decompression algorithm. (8) In addition, multiple acceleration cores can be implemented on one DPU acceleration card to realize parallel decompression of multiple input compressed data.

[0068] Figure 3 The figure is a decompression implementation step diagram of the DPU acceleration card based on the heterogeneous architecture in an embodiment of the present application. The steps include:

[0069] Step S310: The server sends the compressed data to the DPU acceleration card, or the compressed data is input to the DPU acceleration card through the external interface;

[0070] Step S320: The DPU acceleration card caches the input compressed data into the DDR;

[0071] Step S330: The acceleration core reads the compressed data from the DDR and decompresses it to the original data;

[0072] Step S340: the acceleration core stores the decompressed original data to the DDR;

[0073] Step S350: the server reads the decompressed original data from the DDR of the DPU acceleration card.

[0074] Figure 4 The figure shows the input and output of the acceleration core, the input is compressed data (compressed data stream or sub-data stream), and the output is original data. The process of merging and decompressing multiple data streams is hidden here.

[0075] The snappy decompression function of the heterogeneous platform aims to receive a snappy compressed data stream, parse the tag information from the compressed data stream, and parse the original data (decompressed data) from the data stream according to the tag information. The snappy decompression function is executed on the DPU acceleration card, and we call this function module an acceleration core. In this scheme, the acceleration core can be understood as different function modules.

[0076] Figure 5 The figure shows the input and output of the acceleration core, the input is compressed data (compressed data stream or sub-data stream), and the output is original data. The process of merging and decompressing multiple data streams is hidden here. Figure 5 The method comprises the following steps: step S510, the acceleration core starts processing the data to be decompressed in the heterogeneous architecture; step S520, the acceleration core reads data from the DDR; step S530, the acceleration core splices the data into a sliding window form; step S540, it is judged whether the 10 bytes of the frame header match, if yes, step S550 is entered, if not, step S530 is returned; step S550, decompress the data according to the snappy format; step S560, write the decompressed data back to the DDR; step S570, end the decompression processing flow under the heterogeneous architecture.

[0077] 1. In an embodiment of the present application, the SNAPPY compression format is (the compressed block is represented by chunk), as shown in Figure 6

[0078] 2. A compressed file is composed of multiple chunks.

[0079] 3. Each chunk contains:

[0080] (1) 1 byte of chunk type (i.e. block type information):

[0081] ​① 0xFF represents the first chunk of the file. Only used as an identifier, no actual decompression meaning.

[0082] ② 0x00 represents a normal compressed data chunk.

[0083] (2) Length information N composed of 3 bytes;

[0084] ① The length of the first chunk is fixed at 6;

[0085] ② The length of other chunks is determined by the compressed content.

[0086] (3) N bytes of compressed data content;

[0087] ① The compressed content of the first chunk is fixed as "sNaPpY".

[0088] ② The compressed content of other chunks is the actual compressed data.

[0089] In the initial preparation stage of decompression, the steps include: (1) compressed data is unloaded from the host to the DPU acceleration card, the host sends the compressed data to the DPU acceleration card DDR through the PCIe channel and starts the execution of the acceleration core. Or the compressed data is input to the DDR of the DPU acceleration card through the fiber, network interface, etc., and the DPU acceleration card starts the execution of the acceleration core to decompress. (2) The compressed data is read from the DDR to the acceleration core, and the acceleration core reads the compressed data from the DDR, and the bit width of the data read each time is determined by the pin bit width of the DDR, which can also be customized, which can be one byte or two bytes or more bytes, and we define the data bit width of the DDR as DW (Data Width).

[0090] Figure 7 The flowchart of the pointer indicating the sliding window writing position in an embodiment of the application is shown. Figure 7 The steps shown are used in the process of the acceleration core finding the first chunk of the compressed file. First, after the acceleration core reads the compressed data, it needs to detect whether it is a normal compressed file, and if it is not a normal compressed file, it will not be processed. The detection method is to judge whether the first chunk is correct, as shown in 7, the first chunk is composed of "0xFF" + "0x06" + "sNaPpY", a total of 10 bytes. Since the 10 bytes of the first chunk are all fixed values, in order to reduce the processing time, the first chunk is not parsed, but the content of the first chunk is directly judged. The judgment method adopts a sliding window method.

[0091] Here, a register with a bit width of N1 bytes is defined as a sliding window, N1 needs to meet the following conditions: (1) the sliding window width N1 ≥ DW*2. (2) the sliding window width N1 ≥ 10.

[0092] At the same time, a pointer is defined here to indicate the position that the sliding window can write. The pointer is 0 by default, pointing to the starting position of the sliding window. When reading a DW byte data from the DDR and writing it into the position pointed by the pointer in the sliding window, the pointer value increases by DW. It is necessary to judge whether the remaining width in the sliding window can store the data content of the next reading each time the DDR is read, and the requirement is "N1-pointer value≥DW", that is, the data read and written into the sliding window cannot overflow outside the sliding window. When the pointer value is greater than or equal to 10, it indicates that the sliding window may contain the data of the first chunk, and therefore the first 10 bytes of the sliding window are compared with the data of the first chunk.

[0093] If the comparison with the first chunk does not match, the entire sliding window is shifted left by one byte, and the pointer value is reduced by 1. The above steps are repeated to continue to judge whether the pointer value is greater than or equal to 10, and if it is greater than or equal to 10, the comparison is continued.

[0094] If the comparison with the first chunk matches, it indicates that the starting position of the compressed file is found, the sliding window is shifted left by 10 bytes, which is equivalent to discarding the data of the first chunk. Then it is judged whether the pointer value in the sliding window is greater than DW, and if it is greater than DW, the data of the first DW bytes in the sliding window is read out and sent to the subsequent module for decompression.

[0095] Through the above operation, the comparison of the first chunk is completed, the starting position of the file is found, and the decompression from the beginning of the file is ensured.

[0096] Figure 8 The extraction and output flow of the data content in a compressed block in an embodiment of the application is shown. Figure 8 The method steps shown are used in the unpacking process of the compressed blocks of the compressed file except the first compressed block by the acceleration core.

[0097] From the second chunk, they are all standard compressed formats, and before decompression, the format of each chunk needs to be unpacked first. The package format is: (1) one byte of chunk type, from the second chunk, the chunk type is all 0x00; (2) three bytes of chunk length information N; (3) N bytes of chunk data content.

[0098] The design also adopts the sliding window mode, and the main purpose is to extract the N byte chunk data content from the chunk format and transmit it to the next module for decompression. The sliding window width is N2 bytes, and N2 needs to meet: (1) N2≥4; (2) N2≥DW*2.

[0099] The sliding window is also provided with a pointer for indicating the position that the sliding window can write. The upper module inputs one DW byte width of data each time, and the pointer value is increased by DW each time after inputting one data, and points to the latest position of the sliding window. When the pointer value is greater than or equal to 4 after inputting one or more data, the length information of the chunk is extracted from the B2-B4 positions of the sliding window. At the same time, the sliding window is shifted left by 4 bytes, and the pointer value is reduced by 4, and the first data of the sliding window is the "data content" of the current chunk.

[0100] Since the upper module continuously writes data into the sliding window of the module, after extracting the length information, it is judged each clock cycle whether the value of the pointer is greater than or equal to DW, if yes, the first DW bytes of the sliding window are output to the lower module for decompression, and the pointer value is reduced by DW, and the extracted length value is also reduced by DW. When the length value is reduced to less than or equal to DW, it is indicated that it is the last data of the current chunk, and the data corresponding to the length value is output when the remaining data in the sliding window is greater than or equal to the length value, thus completing the extraction and output of the data of one chunk.

[0101] Figure 9 The structure diagram of the data content compressed by the embodiment of the application is shown. Figure 9 The method steps shown are used for parsing the compressed chunk data content (i.e., chunk_body) contained in the compressed chunk. Specifically, it includes:

[0102] After the above processing, the chunk data content of N bytes in total is sent to the module, i.e., the chunk_body part in the following figure. The chunk_body part includes: (1) a CRC32 part of four bytes, CRC32 is a cyclic redundancy check algorithm, which is used for checking of transmission data, and is mainly used for detecting errors in the data transmission or storage process. (2) length information encoded by Varint, which occupies 1-3 bytes, and varint encoding is a method of serializing integers using one or more bytes, which encodes the integer as a variable-length byte to achieve compression. (3) the remaining part is the compressed data part, and the compressed data adopts a tag method.

[0103] Here, the sliding window and the pointer are also adopted, and the uncompressed length and the compressed data part are parsed from the chunk_body. A sliding window is defined, and the sliding window width is N3 bytes, and N3 needs to satisfy: (1) N3≥7; (2) N3≥DW*2.

[0104] Figure 10The flow chart for parsing the length information of the varint encoding in an embodiment of the present application. When the pointer value in the sliding window is greater than or equal to 4, the data in the sliding window is left shifted by 4 bytes, and the CRC content of the first 4 bytes is discarded. After the left shift by 4 bytes, the fifth byte becomes the first data in the sliding window, and then the parsing of the length information in the varint encoding format is started. The varint encoding format is defined as follows: (1) a varint data is composed of one or more bytes, and each byte has 8 bits; (2) the highest bit of each byte is 1, which indicates that the next byte also belongs to the current data; (3) the highest bit of each byte is 0, which indicates that the current byte is the last byte of the current data; (4) the low 7 bits of each byte of the current data are spliced, and the new data obtained is the actual data represented by the varint; and (5) in the snappy compression, the length information is not more than 3 bytes, and thus the total number of bytes of the varint is not more than 3.

[0105] Figure 11 The processing sequence for parsing the length information of the varint encoding in an embodiment of the present application is shown in the diagram. First, the first byte B5 of the sliding window is read, and if the highest bit of B5 is 0, it indicates that the next byte B6 is also the data of the varint, and the second byte B6 is read again. If the highest bit of B6 is 0, it indicates that the next byte B7 is also the data of the varint. Since the number of bytes is not more than 3, the highest bit is not judged when B7 is read. When the highest bits of B5 and B6 are judged, if the highest bit is 1, it indicates that the varint encoding is ended, and the next byte is not judged.

[0106] Figure 12 The flow for parsing the varint encoding by splicing in an embodiment of the present application is shown in the diagram. In the above processing of the varint encoding, a 21-bit length register is defined, which is used to store the parsed length information uncompressedlength (i.e. the length information). After each varint byte is read, the low 7 bits of the corresponding byte are spliced to the length register. The splicing method is as follows: (1) the initial value of the length register is 0; (2) after the first byte B5 of the sliding window is read, the low 7 bits of B5 are put into the low 7 bits of the length register; (3) the highest bit b7 of the first byte B5 is judged: ① if b7 is 0, it indicates that the next byte B6 is not the data of the varint, and thus B6 is not read. The processing of the varint encoding is completed. The high 14 bits of the length register are 0, and the low 7 bits are composed of b7-b0 of B5. ② if b7 is 1, it indicates that the next byte B6 is also the data of the varint, and B6 needs to be processed.

[0107] In the above parsing varint encoding process, the position of varint encoding end is determined. When the highest bit of the first byte B5 is 0, it indicates that the current varint has only one byte, and the varint encoding ends at the first byte, so the pointer of the sliding window is left shifted by 1 byte. Similarly, when the varint ends at the second or third byte, the pointer of the sliding window is left shifted by 2 or 3 bytes. So that the compressed data (i.e. decompressed data, based on a compressed block to obtain a decompressed data, and the decompressed data is merged to obtain a raw data) is at the first byte.

[0108] So far, the first byte of the sliding window starts as compressed data (i.e. compressed data, included in the block data content). It is necessary to decompress the raw data from the compressed data. The compressed data compresses the data in the form of tags. There are two forms of tags: (1) Literal tag: indicates the next specified length of data that has not appeared in the previous data, and these data can be directly output as decompressed data. The Literal tag is divided into five forms, the main difference is how many bytes are used to represent the length information. (2) Copy tag: indicates that the subsequent specified length of data has appeared in the previously output data, and provides the offset position of the data, which needs to be extracted from the previous data and output as decompressed data. The Copy tag is divided into three forms, the main difference is that the length information and offset position use multiple bytes to represent. The specific format of the tag is shown in Figure 13

[0109] ​The way to distinguish the Literal tag and Copy tag is through the lower 2 bits of the first byte of the tag, i.e. "type" in the following figure. When type = 00, it means the current tag is literal tag. When type = 01, it means the current tag is copyl tag. When type = 10, it means the current tag is copy2 tag. When type = 11, it means the current tag is copy3 tag. The tag is defined as follows: (1) When type = 0, it is literal tag, and when Bit[7:2] of the tag = 60, a following one byte indicates the length of data. It means that the data with the corresponding length after the tag and the length information is directly output as decompressed data. (3) Similarly, when Bit[7:2] of the tag = 60 or 61 or 62, 2, 3 or 4 following bytes are length information, indicating that the data with the corresponding length after the tag and the length information is directly output as decompressed data. (4) When type = 01, it is copyl tag, Bit[4:2] plus 4 is the length information length of data, and Bit[7:5] and a following one byte form an 11-bit offset position offset. It means that the length data has appeared in the previously output data, and the position of the appearance is offset bytes from the current position. (5) When type = 10, it is copy2 tag, Bit[7:2] plus 1 is the length information length of data. The following 2 bytes form the offset position offset. It means that the length data has appeared in the previously output data, and the position of the appearance is offset bytes from the current position. (6) When type = 11, it is copy3 tag, Bit[7:2] plus 1 is the length information length of data. The following 4 bytes form the offset position offset. It means that the length data has appeared in the previously output data, and the position of the appearance is offset bytes from the current position.

[0110] Figure 14The schematic diagram of the write address register and the read address register in an embodiment of the present application. As described above, the copy tag indicates that the next data to be output has appeared in the previously output data, so a RAM memory is designed to cache the previously output data, and when the copy tag is received, the corresponding data is indexed from the RAM memory according to the offset and length information of the copy tag and is output. Therefore, this section mainly defines a RAM memory with a bit width of DW and a total capacity of no less than 64 KB. Meanwhile, a write address register and a read address register are defined. The write address register is mainly used to record the address of the written data, and the read address register calculates the address of the read data from the RAM according to the offset and length information of the copy tag.

[0111] The unit of the write address and the read address is byte, but the transmission bit width of the data and the bit width of the RAM are DW, and when DW is greater than 1 byte, the data written in the RAM may not be an integer multiple of DW, so a temp register is designed to splice the data to DW width and then write the data into the RAM. The bit width of the temp register is DW*2, which is consistent with the sliding window effect described in the previous section.

[0112] When the data is written into the RAM, it is first determined whether the value of the write address register is an integer multiple of DW, and if yes, the data is directly written into the RAM. If not, the data in the RAM corresponding to the write address register is first read and written into the temp register, and then the data to be written is also written into the temp register for bit splicing. When DW bits are spliced on the temp register, the DW-bit data on the temp register is written into the RAM, the temp register is shifted to the right by DW bytes, and the RAM address is incremented.

[0113] Figure 15 The principle schematic diagram of the read RAM register in an embodiment of the present application. When the RAM is read, the read address is calculated, and the calculation method is “read address = write address-offset-length”. After the read address is calculated, the data with the corresponding length length is read from the corresponding RAM and is output.

[0114] In the last scheme, the decompressed data is spliced and output. Figure 16 and Figure 17As shown, the decompressed data comes from two parts: one is the decompressed data of the literal tag, which is directly extracted from the compressed data. The other is the copy tag data, which is extracted from the RAM. The two data ultimately needs to be combined into a data output as decompressed data. Therefore, this section designs a multiple selection switch and a sliding window to splice the two data and output.

[0115] Figure 17 The principle diagram of merging multiple data by using a sliding window in an embodiment of the present application is shown. The sliding window is designed in the same way as described above, and a register with a bit width of DW*2 is used to achieve this. A pointer register is also designed to record the position where data can be written into the sliding window. The data after multiple selection is written into the position indicated by the pointer in the sliding window. When the data in the sliding window is sufficient for DW bytes, the data of the lower DW bytes is output. The output data is written into the DDR for caching. At the same time, the pointer is reset by DW bytes to indicate the latest writable position.

[0116] Another aspect of the present application provides a decompression implementation device based on a DPU acceleration card under a heterogeneous architecture. The DPU acceleration card and the CPU server establish a connection to form a heterogeneous system. The DPU acceleration card includes a DDR storage unit and an acceleration core. The device includes: the DDR storage unit, which is used to receive and store the compressed data stream unloaded by the CPU server through the established connection; the DDR storage unit is used to read the sub-data stream with a preset byte length in the compressed data stream into the acceleration core in sequence; and the acceleration core is used to decompress the sub-data stream. The compressed data stream includes a compressed file, and the compressed file is composed of multiple compressed blocks. The compressed blocks carry block type information. The acceleration core is used to decompress the sub-data stream, specifically including: comparing and searching for the starting position of the compressed file in the sub-data stream by using a sliding window, and the starting position of the compressed file is identified by the block type information; for the compressed blocks constituting the compressed file, the decompressed data contained in each compressed block in the compressed data stream is obtained by parsing according to a preset decompression mode; and the decompressed data contained in each compressed block obtained by decompression is merged into a raw data. The DDR storage unit is also used to write back the raw data obtained after decompression by the acceleration core, and transmit the raw data back to the CPU server through the established connection, to realize the decompression from the compressed data stream to the raw data under the heterogeneous system.

[0117] In some embodiments of the present application, the sliding window register is pre-configured in the decompression implementation device based on the DPU acceleration card under the heterogeneous architecture for the acceleration core to decompress the sub-data stream, and the sliding window register comprises: a first sliding window register, which is used to find the position of the first compressed block of the compressed file in the sub-data stream by using the sliding window method, so as to determine the starting position of each compressed file; wherein each compressed block further contains block length information and block data content; a second sliding window register, which is used to parse the block data content from the compressed block according to the block length information by using the sliding window method, the block data content contains length information and compressed data, and the compressed data meeting the length information is read from the block data content; the decompressed data contained in each compressed block is decompressed from the compressed data by using the label contained in the compressed data; and a third sliding window register, which is used to merge the decompressed data contained in each compressed block obtained by decompression based on different labels into one piece of original data by using the sliding window method and a multiple-to-one switch.

[0118] The method and device provided by the present application are applied to the snappy decompression scene in the field of databases and big data, can unload the snappy decompression work from the original execution on the CPU to the execution on the heterogeneous system, widen the implementation scheme of the snappy decompression function, and have the following improvements compared with the original technology:

[0119] (1) The architecture of the snappy decompression function is customized in the heterogeneous system, the snappy decompression function under the heterogeneous architecture is realized through a brand-new design scheme, and the rate of snappy decompression is greatly improved.

[0120] (2) After the task is unloaded to the heterogeneous system, the CPU only needs to send some control instructions, and no longer needs to perform heavy computing tasks. The CPU occupancy is released, and space is provided for the CPU to process other tasks.

[0121] (3) The heterogeneous architecture on which the acceleration core relies, through which the data interaction between the CPU and the DPU acceleration card is realized.

[0122] The present application can be applied to the decompression in the field of databases and big data, mainly by realizing the snappy decompression algorithm and the like on the heterogeneous platform, while the prior art scheme relies on the CPU to call the snappy decompression function to realize the process.

[0123] Those of ordinary skill in the art will appreciate that the various illustrative components, systems and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or both. The particular implementation is dependent on the specific application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, for example, the hardware can comprise an electronic circuit, an Application Specific Integrated Circuit (ASIC), a suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the application are the program or code segments to perform a specific task. The program or code segments can be stored in a machine-readable medium, or transmitted by a carrier wave as data signals over a transmission medium or communication link.

[0124] It is to be understood that the application is not limited to the particular configurations and processes described herein and shown in the drawings. For simplicity, detailed descriptions of known methods and apparatuses are omitted so as not to obscure the disclosure. In the above-described embodiments, several specific steps are described and illustrated as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and the order of the steps can be changed, or other steps can be added, or replaced, or eliminated, depending on the application.

[0125] In the present application, features described and / or illustrated in relation to one embodiment can be used in the same or a similar way in one or more other embodiments, and / or combined with or instead of features of other embodiments.

[0126] The above description is only preferred embodiments of the present application, and is not intended to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A decompression implementation method based on a DPU acceleration card under a heterogeneous architecture, characterized in that, A DPU acceleration card and a CPU server establish a connection to form a heterogeneous system, the DPU acceleration card comprising a DDR storage unit and an acceleration core, and the method comprises the following steps: The DPU acceleration card receives a compressed data stream offloaded by the CPU server through the established connection and stores the compressed data stream on the DDR storage unit of the DPU acceleration card; wherein the compressed data stream comprises a compressed file, the compressed file is composed of multiple compressed blocks, the compressed blocks carry block type information, and each compressed block further comprises block length information and block data content; A sub-data stream of a preset byte length in the compressed data stream is read from the DDR storage unit into the acceleration core in sequence, and the sub-data stream is decompressed by the acceleration core; The step of decompressing the sub-data stream by the acceleration core comprises: comparing and searching for the start position of the compressed file in the sub-data stream by using a sliding window, and the start position of the compressed file is identified by the block type information; for the compressed blocks constituting the compressed file, the decompressed data contained in each compressed block in the compressed data stream is obtained by parsing according to a preset decompression mode; and the decompressed data contained in each compressed block is merged into a raw data stream; The raw data obtained after decompression by the acceleration core is written back to the DDR storage unit, and the raw data is transmitted back to the CPU server through the established connection; The step of obtaining the decompressed data contained in each compressed block in the compressed data stream by parsing according to a preset decompression mode comprises: parsing the block data content from the compressed block according to the block length information by using a sliding window, the block data content comprises length information and compressed data, and the compressed data conforming to the length information is read from the block data content; and the decompressed data contained in a single compressed block is decompressed from the compressed data by using the tag contained in the compressed data; The types of the tag include a literal tag and a copy tag; the step of decompressing the decompressed data contained in a single compressed block from the compressed data by using the tag contained in the compressed data comprises: when the tag type is a literal tag, directly taking the data of a specified length in the compressed data as the decompressed data, and using the compressed block with the literal tag to store non-repeated information; and when the tag type is a copy tag, copying the data of a specified length from the previous data according to the offset position contained in the compressed data as the decompressed data, and using the compressed block with the copy tag to store information that exists repeatedly.

2. The method of claim 1, wherein, The step of comparing and searching for the start position of the compressed file in the sub-data stream by using a sliding window comprises: comparing and searching for the position of the first compressed block of the compressed file in the sub-data stream by using a sliding window, so as to determine the start position of each compressed file.

3. The method of claim 1, wherein, The step of merging the decompressed data contained in each compressed block into a raw data stream comprises: merging the decompressed data contained in each compressed block obtained based on different tags into a raw data stream by using a multiple-to-one switch.

4. The method of claim 3, wherein, The block data content further comprises a cyclic redundancy check bit for detecting or checking decoding data errors; The length information contained in the block data content is encoded using the rules of varint encoding.

5. The method of claim 3, wherein, The copy tag is further divided into multiple copy forms according to length information and offset positions, and the compressed data is identified by using a copy form keyword; The previous data is cached by using a pre-designed RAM memory, a read address register and a write address register on the DPU acceleration card for reading and copying.

6. The method of claim 1, wherein, The preset decompression mode includes any one of snappy decompression, Gzip decompression, LZ4 decompression and Brotli decompression. The DPU acceleration card and the CPU server are connected through a PCIe channel or a network interface.

7. A decompression implementation device based on a DPU accelerator card under a heterogeneous architecture, characterized in that, The DPU acceleration card and the CPU server form a heterogeneous system by establishing a connection, and the DPU acceleration card includes a DDR storage unit and an acceleration core. The DDR storage unit is configured to receive and store a compressed data stream offloaded by the CPU server through the established connection, sequentially read a sub-data stream of a preset byte length in the compressed data stream to the acceleration core from the DDR storage unit, and perform decompression processing on the sub-data stream by the acceleration core; wherein the compressed data stream includes a compressed file, the compressed file is composed of multiple compressed blocks, each compressed block carries block type information, and each compressed block further includes block length information and block data content; The acceleration core is configured to perform decompression processing on the sub-data stream, specifically including: comparing and searching for a start position of the compressed file in the sub-data stream by using a sliding window mode, and the start position of the compressed file is identified by the block type information; for the compressed blocks constituting the compressed file, the decompression data contained in each compressed block in the compressed data stream is obtained by decompression according to the preset decompression mode; the decompression data contained in each compressed block obtained by decompression is merged into one piece of original data; the step of obtaining the decompression data contained in each compressed block in the compressed data stream according to the preset decompression mode includes: obtaining the block data content from the compressed block according to the block length information by using the sliding window mode, the block data content includes length information and compressed data, and reading the compressed data meeting the length information from the block data content; decompressing the decompression data contained in a single compressed block from the compressed data by using the tag contained in the compressed data; the type of the tag includes a literal tag and a copy tag; the step of decompressing the decompression data contained in a single compressed block from the compressed data by using the tag contained in the compressed data includes: when the tag type is the literal tag, directly taking the data of a specified length in the compressed data as the decompression data, and using the compressed block with the literal tag to store non-repeated information; when the tag type is the copy tag, copying the data of a specified length from the previous data as the decompression data according to the offset position contained in the compressed data, and using the compressed block with the copy tag to store information that exists repeatedly. The DDR storage unit is further configured to write back the original data obtained after decompression by the acceleration core, and transmit the original data back to the CPU server through the established connection.

8. The apparatus of claim 7, wherein, In the decompression implementation device based on the DPU acceleration card under the heterogeneous architecture, a sliding window register is pre-set for the acceleration core to perform decompression processing on the sub-data stream, and the sliding window register includes: a first sliding window register for searching the position of the first compressed block of the compressed file in the sub-data stream by using the sliding window method, so as to determine the starting position of each compressed file; wherein each compressed block further comprises block length information and block data content; a second sliding window register for parsing the block data content from the compressed block according to the block length information by using the sliding window method, wherein the block data content comprises length information and compressed data, reading the compressed data conforming to the length information from the block data content, and decompressing the decompressed data contained in each compressed block from the compressed data by using the tag contained in the compressed data; a third sliding window register for merging the decompressed data contained in each compressed block obtained based on different tags into one original data by using the sliding window method and a multiplexing switch.

Images