Data compression in data conversion accelerator

Abstract

A method includes obtaining, by a data conversion accelerator, a first command and first source data associated with the first command. The method also includes generating, by the data conversion accelerator, one or more containers using the first command. Each of the one or more containers may have a container size and may be used to store compressed source data. The method also includes obtaining an operation mode of the data compression operation. The method also includes performing a data compression operation on a first portion of the first source data using the mode of operation to obtain the compressed source data. The method also includes storing the compressed source data in a first container of the one or more containers.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 513,413, filed on July 13, 2023, entitled “DATA COMPRESSION IN A DATATRANSFORMACCELERATOR,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to data conversion acceleration and, more particularly, to data compression in a data conversion accelerator. Background Art

[0004] Unless otherwise indicated herein, the materials described herein are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0005] A data conversion accelerator is a coprocessor device used to accelerate data conversion operations of various applications (e.g., data analysis applications, big data applications, storage applications, encryption applications, and network applications). For example, a data conversion accelerator can be configured as a storage accelerator and / or an encryption accelerator.

[0006] The subject matter claimed in this disclosure is not limited to implementations that solve any disadvantages or operate only in environments such as described above. Rather, this background is provided merely to illustrate one example technology area in which some implementations described in this disclosure may be practiced. Summary of the invention

[0007] In an example embodiment, a method may include obtaining, by a data conversion accelerator, a first command and first source data associated with the first command. The method also includes generating, by the data conversion accelerator, one or more containers using the first command. Each of the one or more containers may have a container size and may be used to store compressed source data. The method also includes obtaining an operating mode of a data compression operation. The method also includes performing a data compression operation on a first portion of the first source data using the operating mode to obtain the compressed source data. The method also includes storing the compressed source data in a first container of the one or more containers.

[0008] The objects and advantages of the embodiments will be realized and attained by at least the elements, features, and combinations particularly pointed out in the claims.

[0009] Both the foregoing general description and the following detailed description are exemplary and illustrative rather than restrictive of the invention as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings will provide a more specific and detailed description and explanation of the exemplary embodiments:

[0011] Figure 1 A block diagram illustrating an example environment for data compression in a data conversion accelerator is shown;

[0012] Figure 2A An example of a compression operation in a data conversion accelerator is shown;

[0013] Figure 2B An example of a compression operation in a data conversion accelerator is shown;

[0014] Figure 3 A flow chart illustrating an example method of data compression in a data conversion accelerator; and

[0015] Figure 4 An example of a computing device is shown. DETAILED DESCRIPTION

[0016] The data conversion accelerator can be used as a co-processor device with a host device to accelerate data conversion operations for various applications (e.g., data analysis, big data, storage, and / or network applications). The data conversion operations may include, but are not limited to, compression, decompression, encryption, decryption, authentication tag generation, authentication, data deduplication, non-volatile memory express (NVMe) protection information (PI) generation, NVMe PI verification, and / or real-time verification.

[0017] Many file systems and / or storage systems use block-like structures, e.g., physical disks organized into blocks and / or sectors. Data in a file system may be managed in units of blocks (e.g., a typical block size may be 1KB, 2KB, 4KB, and / or 16KB), and the block size may be determined during the initialization of the file system. Sometimes, a program utilizing a file system may use a determined block size and / or may perform storage operations on raw sectors and / or blocks of a storage system. In such cases, performance degradation may occur because storage in the file system may not be performed in units of blocks defined in the file system or storage system.

[0018] Alternatively or additionally, erasing stored data and / or rewriting new data may be performed by programs that may or may not be performed in units of blocks defined in the file system. In this case, data storage in the file system may become increasingly fragmented, where blocks of different sizes may be used for various operations. For example, a heap used for memory management may be fragmented and / or performance may be degraded in a situation where a first block is written, later erased, and replaced by a new block, and each of the aforementioned blocks has a different size.

[0019] At least some aspects of the present disclosure address these and other disadvantages by determining and generating a uniform container size to store compressed source data. As the source data is compressed, the compressed source data can be stored in a uniformly sized container so that all blocks in the file system (as described above) are the same size. In this way, memory usage can be improved because memory fragmentation can be reduced, load time from memory can be increased, indexing of the file system can be improved, and / or performance tuning of the file system can be achieved.

[0020] The data conversion operations performed by the data conversion accelerator can be separated based on the direction associated with the data, such as the encoding direction (associated with transmitting and / or encoding data) and the decoding direction (associated with receiving and / or decoding encoded data). For example, the encoding direction data conversion operation may include NVMe PI verification of the input data, compression of the encoded data, deduplication hash generation, padding, encryption, encrypted hash generation and NVMePI generation, and real-time verification of the encoded data and / or a combination of one or more of the aforementioned encoding direction data conversion operations. In another example, the decoding direction data conversion operation may include decryption (e.g., with or without a cryptographic hash for authentication), de-stuffing, decompression, hash generation for data verification, NVMe PI verification of the decoded data and / or a combination of one or more of the aforementioned decoding direction data conversion operations.

[0021] The data conversion accelerator may include various data conversion engines that may be configured in a pipeline to perform various data conversion operations in an encoding direction or a decoding direction. For example, a first pipeline may include a first arrangement of data conversion engines and may be used to perform encoding direction data conversion operations. In another example, a second pipeline may include a second arrangement of data conversion engines and may be used to perform decoding direction data conversion operations.

[0022] In some cases, data conversion operations performed in the encoding direction (e.g., performed by a data conversion engine included in a first pipeline) can generate metadata and / or other outputs that can be used by a second pipeline and / or an associated data conversion engine configured to perform data conversion operations in the decoding direction. For example, according to some aspects of the present disclosure, at least some outputs from encoding direction data conversion operations can generate metadata that can be used in a decoding direction pipeline, which metadata can facilitate data conversion operations performed in the decoding direction pipeline. In this case, data conversion operations performed in the decoding direction (e.g., data conversion operations performed by a data conversion engine in a pipeline in the decoding direction) can reduce the latency of data conversion operations in the decoding direction and / or can increase the throughput of data conversion operations in the decoding direction.

[0023] Figure 1 A block diagram of an example environment 100 for data compression in a data conversion accelerator 120 according to at least one embodiment of the present disclosure is shown. The environment 100 may include an external device 110 and a data conversion accelerator 120. The external device 110 may include an external processor 112 and an external memory 114. The data conversion accelerator 120 may include an internal processor 122, an internal memory 124, and a data conversion engine 126. The data conversion engine 126 may include at least a data compression engine 128.

[0024] In some embodiments, the external device 110 (e.g., a host, a host server, etc.) can communicate with the data conversion accelerator 120 via a data communication interface (e.g., a Peripheral Component Interconnect Express (PCIe) interface, a Universal Serial Bus (USB) interface, and / or other similar data communication interfaces). In some embodiments, when a user requests to convert source data that may be located in the external memory 114, software (e.g., a software driver) on the external device 110 and operated by the external processor 112 can be directed to generate metadata about converting the source data in the external memory 114 (e.g., but not limited to, data conversion command pre-data including command descriptions, descriptor lists that dereference different parts of the metadata, descriptor lists that dereference source data and target data buffers, command pre-data including conversion algorithms and related parameters, source and action tokens that describe different parts of the source data and conversion operations applied to the different parts, and / or additional command metadata). In some embodiments, the software can generate the metadata in the external memory 114 based on the source data that can be obtained from one or more sources. For example, the source data may be obtained from a memory associated with the external device 110 (e.g., a storage device), a buffer associated with the external device 110, a data stream from another device, etc. In these and other embodiments, obtaining the source data may include copying or moving the source data to the external memory 114.

[0025] In some embodiments, the software may instruct the external processor 112 to generate metadata associated with the source data. In some embodiments, the metadata may be stored in one or more input buffers. For example, in the case where the metadata includes a data conversion command that may include a list of source descriptors, destination descriptors, command pre-data, source tokens and action tokens, and additional command metadata, each individual component of the metadata may be stored in a separate input buffer (e.g., the data conversion command is in a first input buffer, the pre-data is in a second input buffer, the source token and the action token are in a third input buffer, and so on). In some embodiments, the input buffer associated with the metadata may be located in the external memory 114. Alternatively or additionally, the input buffer associated with the metadata may be located in the internal memory 124. Alternatively or additionally, the input buffer may be located in the external memory 114 and the internal memory 124. For example, one or more input buffers associated with the metadata may be located in the external memory 114, and one or more input buffers associated with the metadata may be located in the internal memory 124. In these and other embodiments, the external processor 112 may instruct the software to retain one or more output buffers, which may be used to store outputs from the data conversion accelerator 120. In some embodiments, the output buffer may be located in the external memory 114. In some embodiments, the output buffer may be located in the internal memory 124 of the data conversion accelerator 120.

[0026] In the case where the software instructs the external processor 112 to generate metadata and store the metadata in the internal memory 124 (e.g., in an input buffer located in the internal memory 124) and / or in the external memory 114, the external processor 112 may send a command to the data conversion accelerator 120 (e.g., to a component of the data conversion accelerator 120, such as the internal processor 122) through the data communication interface. For example, the internal memory 124 may be accessed and / or addressed by the external processor 112 via the data communication interface, and in the case where the data communication interface is PCIe, the internal memory 124 may be mapped to the address space of the external device 110 using a base address register associated with an endpoint of the PCIe (e.g., the data conversion accelerator 120).

[0027] In some embodiments, the software may instruct the data conversion accelerator 120 to process the data conversion command. For example, the software may instruct the data conversion accelerator 120 to obtain an address that may point to the data conversion command. In some embodiments, the data conversion command may be used by the data conversion accelerator 120 to convert the source data based on the data conversion operation included in the data conversion command. In some embodiments, the data conversion operation indicated by the data conversion command may be performed by the data conversion engine 126. In some embodiments, the data conversion engine 126 may be arranged according to the data conversion command and / or metadata (e.g., metadata stored in the external memory 114 and / or stored in the internal memory 124) so ​​that the data conversion engine 126 forms a data conversion pipeline that can be configured to perform data conversion operations on the source data.

[0028] In some embodiments, the address and / or the data conversion command may be located in the external memory 114. In this case, the data conversion accelerator 120 (e.g., the internal processor 122 and / or the data conversion engine 126) may obtain the address and / or may use the data communication interface to access the data conversion command in the external memory 114. Alternatively or additionally, the address and / or the data conversion command may be located in the internal memory 124, and the address may be obtained by the internal processor 122 and / or the data conversion engine 126.

[0029] In these and other embodiments, the external device 110 may send metadata to the data conversion accelerator 120 using the data communication interface, the internal processor 122 may instruct to store the metadata in the internal memory 124, and the internal processor 122 may return the address of the stored metadata to the external processor 112. Alternatively or additionally, the external device 110 may transfer the metadata directly to the internal memory 124 of the data conversion accelerator 120 using the data communication interface.

[0030] In some embodiments, data conversion operations performed by the data conversion engine 126 (e.g., one or more data conversion operations performed in a data conversion pipeline as described herein) can generate second metadata that can be used to configure a second pipeline in the data conversion accelerator 120, as described herein.

[0031] The data conversion accelerator 120 can be used to perform data conversion operations using one or more pipelines, which include a configuration of the data conversion engine 126. The pipelines in the data conversion accelerator 120 can be described as performing data conversion operations in at least two directions (encoding direction and / or decoding direction). The encoding direction data conversion operations performed by the first pipeline in the data conversion accelerator 120 may include one or more of the following: NVMe PI verification of input data, compression of encoded data, deduplication hash generation, padding, encryption, encrypted hash generation and NVMe PI generation, and / or real-time verification of encoded data. The decoding direction data conversion operations performed by the second pipeline in the data conversion accelerator 120 may include one or more of the following: decryption of input data and / or converted data (e.g., data obtained from input data) (e.g., with or without verification of input data and / or converted data generation), de-stuffing, decompression, de-duplication hash generation and / or NVMe PI verification of encoded data.

[0032] In some embodiments, the data conversion accelerator 120 is operable to support multiple data conversion sessions, wherein the data conversion session may include source data, associated metadata, and a data conversion engine 126 (e.g., the data conversion engine 126 is arranged in a data conversion pipeline), as described herein. In some embodiments, one or more data conversion commands may include the same or similar algorithms in the data conversion operation. In this case, separate data conversion commands may be grouped together into a data conversion session. In some embodiments, multiple data conversion commands grouped into a data conversion session may include the same or similar metadata. In this case, the data conversion accelerator 120 may store the source data and / or metadata in an internal memory 124, as described herein, and the data conversion accelerator 120 may provide the address to the external device 110 so that the external device 110 may include the address in the data conversion command belonging to the session.

[0033] In these and other embodiments, one or more source descriptors may be included in the plurality of data conversion commands, which may point to one or more input buffers that may be configured to store metadata shared between multiple commands in a session. Alternatively or additionally, the plurality of data conversion commands may include one or more source descriptors, which may point to one or more input buffers that may be configured to store source data and / or metadata that are unique to different commands of the session. In the case where a first data conversion command and a second data conversion command have the same input data and / or metadata, the corresponding source descriptors may point to the same one or more input buffers. In the case where the first data conversion command and the second data conversion command have different input data and / or metadata, the corresponding source descriptors may point to different input buffers if appropriate.

[0034] For example, a first data conversion session may include a first data conversion command, which may include first source data in a first input buffer and associated first metadata stored in a second input buffer of the internal memory 124, and one or more data conversion engines 126 may be arranged in the first data conversion pipeline. The first data conversion session may include a second data conversion command, which may include first source data (stored in the first input buffer) and first metadata. Alternatively or additionally, the second data conversion command may utilize shared source data and / or shared metadata to perform a data conversion operation. In an example, the first data conversion command and the second data conversion command may include the same source descriptor as the first data conversion command, and the second data conversion command may use the same source data and / or metadata stored in the corresponding input buffer.

[0035] The data conversion accelerator 120 and / or the components included therein (e.g., the internal processor 122, the internal memory 124, and / or the data conversion engine 126) may be implemented using various systems and / or devices. For example, the data conversion accelerator 120 may be implemented using hardware, software, firmware, a field programmable gate array (FPGA), a graphics processing unit (GPU), and / or any combination of the above embodiments.

[0036] The data compression engine 128 may be used to perform data compression operations in the data conversion accelerator 120. The data compression operations may include one or more data compression algorithms, all of which may be performed by the data compression engine 128 of the data conversion accelerator 120. For example, the data compression algorithms may include, but are not limited to, substring search algorithms (which may be included in the data compression engine 128 and / or external devices), dictionary-based matching algorithms (which may replace substrings found in the source data with additional compressed markers), lossless encoding algorithms (which may encode substrings of the source data and / or matching dictionary entries for substrings in the source data), partial match prediction algorithms, and / or other data compression algorithms.

[0037] Where a dictionary-based algorithm is used, the dictionary may be a static pre-initialized dictionary and / or an adaptive dictionary (e.g., the dictionary may be built and / or revised as part of the data compression operation), which may utilize a history buffer that may be included in the data conversion accelerator 120. Some example data compression algorithms may include, but are not limited to, LZ77, LZ78, LZ4, LZS, Deflate, GZIP, ZLIB, XP10, Zstandard, Brotli, Huffman codes, asymmetric digital systems (ANS), Bruns-Wheeler transforms, and / or other data compression algorithms.

[0038] The aforementioned data compression algorithm may include one or more dictionaries, context-based models, codebooks, and / or initial parameters, which may be associated with the data compression algorithm employed by the data compression engine 128. In the present disclosure, unless otherwise specifically stated, references to "dictionary" may refer to dictionaries, context-based models, codebooks, and / or initial parameters, as the aforementioned may be associated with different data compression algorithms that may be implemented by the data compression engine 128.

[0039] In some embodiments, the data compression engine 128 can obtain source data for compression and storage in one or more containers. In some embodiments, the data conversion accelerator 120 can obtain source data, for example, from an external device 110. The source data can be stored in an external memory 114 and / or an internal memory 124. In some cases, the source data can be processed by one or more data conversion engines 126 before the data compression engine 128. Alternatively or additionally, the output of the data compression engine 128 (e.g., a container storing compressed source data, as described herein) can be subjected to additional data conversion operations by the data conversion engine 126 and / or can be stored or transmitted by the data conversion accelerator 120 until the compressed source data stored in the container can be decompressed to obtain the source data. In these and other embodiments, the data conversion accelerator 120 can be used to generate a container using a command, as described herein.

[0040] In some embodiments, the data compression engine 128 may perform data compression operations in view of subsequent data conversion operations that may be performed with respect to the container storing the compressed source data. For example, the data compression engine 128 may reserve a portion of the container (e.g., the compressed source data may fill a first portion of the container and a second portion of the container may be reserved) for NVMePI insertion, padding, encryption (e.g., with or without authentication), dictionary, and / or metadata, which operations may be subsequently performed by one or more additional data conversion engines of the data conversion engine 126, as further described with respect to FIG. 2 .

[0041] In some embodiments, the data conversion accelerator 120 and / or the data compression engine 128 may be used to adjust one or more data compression parameters of the data compression algorithm. In view of the various objectives provided to the data conversion accelerator 120 and / or the data compression engine 128, the data compression parameters may be adjustable. For example, the data compression parameters of the data compression algorithm may be adjusted based on maximization of source data compression, wherein portions of the source data may be compressed to a size smaller than the container size, and fillers may be added to the container so that the container is substantially full (e.g., a combination of compressed source data and fillers may occupy space provided by the container). In another example, the data compression parameters of the data compression algorithm may be adjusted to optimize the size of the compressed source data relative to the container size of the container (e.g., not attempting to improve and / or maximize data compression efficiency, but rather fitting the compressed source data to the container size).

[0042] In some embodiments, the adjustment of the data compression parameters of the data compression algorithm may be performed in response to input (e.g., user input) or in response to results from other data compression operations. For example, in a case where a first data compression of the first source data performed by the data compression engine 128 is greater than the container can store (e.g., overflow), the data compression engine 128 and / or the data conversion accelerator 120 may use the result to adjust the data compression parameters of the data compression algorithm so that a subsequent second data compression of the second source data can be stored in the container (e.g., without overflow). In this case, the data compression engine 128 and / or the data conversion accelerator 120 may consider including the overflow portion of the compressed source data from the first container in the second container. For example, the data compression engine 128 and / or the data conversion accelerator 120 may indicate that the overflow portion of the first compressed source data is included in the second container, and may reduce the amount of the second source data, respectively, so that the overflow portions of the first compressed source data and the second compressed source data may be stored in the second container.

[0043] The data compression engine 128 may perform operations using a determined mode of operation. The mode of operation may be a stateless data compression operation and / or a stateful data compression operation. For example, in a stateless data compression operation, compressed source data (as described herein) stored in each container may be decompressed independently of compressed source data stored in any other container. In another example, in a stateful data compression operation, compressed source data stored in at least some containers may include dependencies on compressed source data stored in one or more previous containers during a decompression operation.

[0044] In the case where the data compression engine 128 uses stateless data compression, a dictionary used in the data compression operation can be initialized for each container (e.g., a static pre-initialized dictionary) so that the compressed source data in each container can be independent of the compressed source data in another container. In this way, the data compression operation performed by the data compression engine 128 using stateless data compression can make the compressed source data stored in the container independent, which can facilitate subsequent more efficient data decompression operations, as described herein.

[0045] In the case where the data compression engine 128 uses stateful data compression, one or more containers (e.g., for storing compressed source data) may include dependencies on subsequent containers. For example, a first container may not include any dependencies (e.g., a dictionary may be initialized using the first container), a second container may include a dependency on the first container (e.g., a dictionary may be updated during the compression of source data into the first container), a third container may include a dependency on the second container (and, by extension, also a dependency on the first container), and so on. In this case, the dependencies may manifest themselves during a data decompression operation. For example, in the case where the second container described above is to be decompressed, the first container may be decompressed first, and then the second container may be decompressed.

[0046] In the case where there is a dependency between at least a first container and a second container (e.g., stateful data compression), the output from the data compression engine 128 associated with the first container may include a frame header, which may include one or more compression parameters associated with the first container (the compression parameters are associated with a data compression operation on the source data stored in the first container), and wherein the frame header may be used for decompression by a subsequent container. For example, in the case where a second container (dependent on the first container) is to be decompressed, the first frame header may be used to first decompress the first container, and then the second container may be decompressed. Each container including subsequent dependent containers may include a frame header that may be used for decompression in the future.

[0047] Alternatively or additionally, a subsequent container (eg, including any container included in the dependent stateful data compression (eg, not the first container)) may include a checksum associated therewith. The checksum may be used to verify the continuity of the compressed source data in the container.

[0048] In these and other embodiments, a plurality of containers (e.g., a plurality of containers in a set) that may include dependencies may be defined, for example, by the data conversion accelerator 120 and / or the external device 110 (e.g., software on the external device 110 and / or the external processor 112). Alternatively or additionally, the number of containers that may be included in a set may be based on user input entered into the external device 110 and / or the data conversion accelerator 120. For example, three containers of a first set may include a first dependency between each other (as described), three containers of a second set may include a second dependency between each other, and so on, where the number of containers included in the set (e.g., three) may be predetermined or defined by user input. The above examples include three set sizes, which are merely exemplary. Any other number of containers may be included in a set (may include one, which may be stateless data compression).

[0049] In some embodiments, the number of containers included in a set can introduce a tradeoff between compression of source data therein and the complexity of decompressing the source data in the container. For example, using stateless data compression (e.g., each container is independent of the other containers), the source data in each container can be decompressed independently, at the expense of lower compression efficiency (e.g., because the dictionary may be static and / or may not be updated based on data compression operations performed on other source data associated with other containers. In another example, using stateful data compression, where the number of containers in a particular set is set to a large value (e.g., such as five or more), the compression efficiency of source data compressed into each subsequent container of the set can be improved (e.g., as the dictionary is updated, the data compression operation can improve), but at the expense of dependencies between containers in the particular set.

[0050] The one or more containers operable to store compressed source data may include a fixed size. Where the container includes a fixed size, the container may provide more efficient storage (e.g., less wasted storage due to fragmentation), faster load times (relative to containers of non-uniform size), better indexing, and / or performance tuning of the memory system. The fixed size of the container may be determined based on one or more considerations, including: an operating system associated with the container and / or the data conversion accelerator 120; software associated with the external device 110, the external processor 112, the data conversion accelerator 120, and / or the internal processor 122; a storage architecture associated with the external memory 114 and / or the internal memory 124; performance tuning of storage operations of the data conversion accelerator 120, the data compression engine 128, and / or the external memory 114 and / or the internal memory 124; and / or user input associated with an estimate of the amount of compression of the source data (e.g., which may indicate the number of containers per command). In these and other embodiments, the fixed size of the container may be determined by the external device 110 and / or the data conversion accelerator 120, and metadata associated with the command may instruct the data compression engine 128 to determine the fixed size of the container.

[0051] In some embodiments, fixed-size containers can facilitate improved memory storage operations because all containers storing compressed source data can be of uniform size, so that the containers can be easily organized (e.g., placement in memory is simplified when all blocks are the same size), and overwriting and / or replacement of containers can be simplified (e.g., when a container is removed, the replacement can simply fill the resulting gap because the replacement container can be substantially the same size as the removed container). An example of compressing source data into one or more containers is further shown and discussed with respect to FIG.

[0052] In some embodiments, the fixed size of the container can be a power of 2, as is commonly used in storage devices. Alternatively or additionally, the fixed size of the container can be any size, not limited to a power of 2. For example, the fixed size of the container can initially be a power of 2, and then an additional amount of space can be included, for example, to accommodate subsequent data conversion operations (such as NVMe PI insertion described herein). In these and other embodiments, the fixed size of the container can be any size supported by the data conversion accelerator 120 and / or a storage device operable to store the container.

[0053] In these and other embodiments, source data obtained by the data compression engine 128 (e.g., obtained via the data conversion accelerator 120) may be submitted to the data conversion accelerator 120 in a command. The command may include at least a source descriptor (pointing to the location of the source data), a destination descriptor (pointing to the location where the container may be output), a data compression algorithm used in a data compression operation in the data compression engine 128, a dictionary associated with the data compression algorithm used by the data compression engine 128, and / or other metadata associated with the data conversion accelerator 120 and / or the source data used by the data compression engine 128. In some embodiments, stateful data compression operations may include sharing dictionaries between commands.

[0054] At the end of the data compression operation associated with the command performed by the data compression engine 128, the data compression engine 128 can be used to generate a report associated with the data compression operation. The report may include the number of consumed bytes of the source data during the data compression operation. Alternatively or additionally, the report may include the compressed data size of the compressed source data during the data compression operation. In these and other embodiments, the data conversion accelerator 120 can be used to obtain the report from the data compression engine 128 and / or transmit the report to a device (e.g., external device 110) associated with the source data. In this way, the external device 110 can be used to determine additional data compression operations (e.g., when a portion of the source data may not be compressed) and / or can maintain an index of the compressed source data to retrieve and decompress at another time.

[0055] In the case where multiple commands include similar data compression algorithms, the multiple commands can be grouped into a session, and the grouped commands can share one or more session parameters. For example, the grouped commands in a session can share the same data compression algorithm, initial data, dictionary and / or encryption key and / or value, etc.

[0056] In some embodiments, the grouped commands in a session may include substantially similar command structures. Alternatively, the first command and / or the last command of the grouped commands may include some differences from other commands in the session, which differences may be attributed to establishing a data compression operation, completing a data compression operation, and / or returning a result associated with a data compression operation (e.g., useful for a stateful data compression operation).

[0057] In some embodiments, a session can be used to support hybrid data compression, which can include a combination of stateful data compression and stateless data compression. For example, some dictionaries can be shared between grouped commands included in a session, while some other components can not be shared. For example, some dictionaries associated with a first command (e.g., a static pre-initialized dictionary and / or a Huffman code book) can be shared with a second command (e.g., a first command and a second command included in a particular session), and a history buffer can be initialized for each command in a particular session (e.g., for use with an LZ77 data compression algorithm).

[0058] In some embodiments, one or more sessions may be used in different scopes. Each of the one or more sessions may belong to a session set associated with a particular scope. For example, a first session set may be used to track data conversion operation grouping commands that may be based on data conversion operation types. A second session set may be used to share the state of a data compression operation.

[0059] In some embodiments, a mapping between two or more scopes may be generated (e.g., by the data conversion accelerator 120 and / or the data compression engine 128). For example, a first session set may include one or more second session sets. In another example, a second session set may be shared when multiple first session sets include the same or similar data compression algorithms and share compression operation states between each other. In some embodiments, there may be a one-to-one mapping between the first session set and the second session set.

[0060] Modifications, additions, or omissions may be made to environment 100 without departing from the scope of the present disclosure. For example, the designation of the various elements in the manner described is intended to help explain the concepts described herein and is not limiting. Furthermore, environment 100 may include any number of other elements or may be implemented in other systems or environments other than the one described. For example, Figure 1 Any components of may be divided into additional components or combined into fewer components.

[0061] Figure 2A and Figure 2B Examples of compression operation 200 and compression operation 250 in a data conversion accelerator according to at least one embodiment of the present disclosure are respectively shown. Compression operation 200 may include first source data 202, second source data 204, first container 206, and second container 208. Compression operation 250 may include first source data 252, container 254, and reserved portion 256.

[0062] like Figure 2A As shown, the compression engine (e.g., Figure 1The data compression engine 128 of the data conversion accelerator may be used to compress at least a portion of the source data for storage in the container. For example, a portion of the first source data 202 may be compressed and stored in the first container 206. In some cases, a portion of the first source data 202 may not be stored in the first container 206 after compression (e.g., the compression amount of the first source data 202 is less than expected). In this case, the portion of the first source data 202 that is not stored in the first container 206 may be returned by the data conversion accelerator and may be included in the second source data 204 for compression and storage in the second container 208, as shown.

[0063] For example, given Figure 1 The data compression engine 128 may compress a portion of the first source data 202 and store the compressed first source data in the first container 206. The data compression engine 128 may return the uncompressed portion of the first source data 202 (e.g., return it to the location where the first source data 202 was sent, such as the internal memory 124 or the external memory 114) for subsequent combination with the second source data 204. The combination of the uncompressed portions of the first source data 202 and the second source data 204 may be compressed by the data compression engine 128 and stored in the second container 208.

[0064] In these and other embodiments, the size of the first container 206 and the size of the second container 208 may be consistent, as determined by the amount of compressed data that can be stored in the first container 206 and the second container 208. For example, as shown, a first amount of source data (e.g., a portion of the first source data 202) compressed and stored in the first container 206 appears larger than a second amount of source data (e.g., a portion of the first source data 202 and the second source data 204) compressed and stored in the second container 208. When compressed, the first amount of source data and the second amount of source data may be substantially consistent because the first amount of source data and the second amount of source data are stored in the first container 206 and the second container 208, respectively, wherein the sizes of the first container 206 and the second container 208 may be substantially consistent.

[0065] Reference now Figure 2B In some embodiments, a portion of container 254 may include a reserved portion 256. Reserved portion 256 may be compressed by a compression engine (e.g., Figure 1 data compression engine 128) and / or a data conversion accelerator (e.g., Figure 1The reserved portion 256 may be reserved in consideration of a subsequent data conversion operation to be performed by the data conversion accelerator (e.g., a data conversion operation after a data compression operation). For example, in some embodiments, the data conversion accelerator may be used to perform a fill operation for a subsequent encryption operation, a generation and insertion operation of an authentication tag in the source data and / or compressed data, and an NVMe PI insertion operation after a data compression operation, and the data conversion accelerator and / or the compression engine may cause a portion of the container 254 to be reserved (e.g., the reserved portion 256) for subsequent filling, encryption, authentication tag generation and insertion, and / or data generation in NVMe PI insertion.

[0066] The reserved portion 256 may or may not be contiguous within the container 254. For example, as shown, the reserved portion 256 may be a block portion of the container 254. In the case where the reserved portion 256 is used for NVMe PI insertion, fragments of the reserved portion 256 may be distributed between portions of the compressed source data, such that the reserved portion 256 may not be contiguous and / or may be distributed within the container 254. For example, a first portion of compressed source data may be stored in the container 254, followed by a first NVMe PI insertion, followed by a second portion of compressed data, followed by a second NVMe PI insertion, and so on.

[0067] Modifications, additions, or omissions may be made to compression operations 200 and 250 without departing from the scope of the present disclosure. For example, the designation of the various elements in the manner described is intended to help explain the concepts described herein and is not limiting. Furthermore, compression operations 200 and 250 may include any number of other elements or may be implemented in systems or environments other than the ones described. For example, Figure 2A and Figure 2B Any components of may be divided into additional components or combined into fewer components.

[0068] Figure 3 A flowchart of an example method 300 for data compression in a data conversion accelerator according to at least one embodiment of the present disclosure is shown. The method 300 may be performed by a processing logic that may include hardware (circuitry, dedicated logic, etc.), software (e.g., software running on a general-purpose computer system or a dedicated machine), or a combination of both, and the processing logic may be included in any computer system or device, such as Figure 1 The data conversion accelerator 120 is configured to:

[0069] For simplicity of explanation, the method described herein is depicted and described as a series of actions. However, the actions according to the present disclosure may occur in various orders and / or simultaneously, and may occur together with other actions not presented and described herein. In addition, not all actions shown may be used to implement the method according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that these methods may alternatively be represented as a series of related states by state diagrams or events. In addition, the methods disclosed in this specification can be stored on articles (e.g., non-transitory computer-readable media) to facilitate the transmission and transfer of these methods to computing devices. The term "article" used herein is intended to include a computer program accessible from any computer-readable device or storage medium. Although illustrated as discrete frames, various frames may be divided into additional frames, combined into fewer frames or eliminated according to the desired implementation.

[0070] At block 302 , a first command and first source data associated with the first command may be obtained by a data conversion accelerator.

[0071] At block 304, one or more containers may be generated by the data conversion accelerator using the first command. In some embodiments, each of the one or more containers may have a container size and may be used to store compressed source data.

[0072] In some embodiments, the container size of the one or more containers can be consistent. The container size can be determined based on at least one of an operating system, platform software, storage architecture, performance tuning, and / or user input. The user input can include an estimate of the number of containers that can be used for each command. Performance tuning can include adjusting data compression parameters, wherein the data compression parameters can be adjusted based on at least one of container size optimization, data compression maximization, and / or user input.

[0073] At block 306, an operating mode of the data compression operation may be obtained. In some embodiments, the operating mode may be stateless data compression. In some embodiments, the data decompression operation on the first container may be independent of the data decompression operation on the subsequent container.

[0074] At block 308, a data compression operation may be performed on the first portion of the first source data. In some embodiments, the data compression operation may use the operation mode to obtain compressed source data.

[0075] At block 310 , compressed source data may be stored in a first container of the one or more containers.

[0076] Without departing from the scope of the present disclosure, modifications, additions, or omissions may be made to method 300. For example, in some embodiments, metadata that may be included in the first command may be used to determine a container size associated with the one or more containers. In another example, a portion of the first container may be reserved for NVMe PI insertion.

[0077] In another example, a second command and second source data associated with the first command may be obtained. The first command and the second command may be included in a session. Alternatively or additionally, a data compression operation may be performed on a second portion of the first source data and a first portion of the second source data using the operation mode to obtain second compressed source data. Alternatively or additionally, the second compressed source data may be stored in a second container of the one or more containers.

[0078] The second container may include one or more dependencies on the first container. The first container may include a header having compression parameters associated with a data compression operation, such that in response to a request to decompress the second container, the first container may be decompressed first and the header may be used to decompress the second container. The first command and / or the second command included in the session may share session parameters, which may include at least a compression algorithm.

[0079] In another example, a report associated with the first source data may be generated. The report may include an amount of the first source data consumed in the data compression operation. Alternatively or additionally, the report may be transmitted to an external device associated with the first source data.

[0080] In another example, the various elements are designated in the described manner to help explain the concepts described herein and are not limiting. In addition, the method 300 may include any number of other elements or may be implemented in other systems or environments other than the described systems or environments.

[0081] Figure 4An example computing device 400 is shown, in which a group of instructions can be executed for making a machine perform any one or more methods discussed herein. Computing device 400 may include mobile phones, smart phones, netbook computers, rack servers, router computers, server computers, personal computers, large computers, notebook computers, tablet computers, desktop computers or any computing devices with at least one processor, etc., in which a group of instructions for making a machine perform any one or more methods discussed herein can be executed. In alternative embodiments, the machine can be connected (for example, networked) to other machines in a LAN, an intranet, an extranet or the Internet. The machine can be operated as a server machine in a client-server network environment. The machine may include a personal computer (PC), a set-top box (STB), a server, a network router, a switch or a bridge or any machine that can perform a group of instructions (sequential or reverse order) specifying the action to be taken by the machine. In addition, although only a single machine is shown, the term "machine" may also include any set of machines that perform a group (or multiple groups) of instructions to perform any one or more methods discussed herein individually or jointly.

[0082] The computing device 400 includes a processing device 402 (e.g., a processor), a main memory 404 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory 406 (e.g., flash memory, static random access memory (SRAM)), and a data storage device 416, which communicate with each other via a bus 408.

[0083] Processing device 402 represents one or more general processing devices, such as a microprocessor, a central processing unit, etc. More specifically, processing device 402 may include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. Processing device 402 may also include one or more special processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. Processing device 402 is configured to execute instructions 426 for performing the operations and steps discussed herein.

[0084] The computing device 400 may also include a network interface device 422 that may communicate with the network 418. The computing device 400 may also include a display device 410 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 412 (e.g., a keyboard), a cursor control device 414 (e.g., a mouse), and a signal generating device 420 (e.g., a speaker). In at least one embodiment, the display device 410, the alphanumeric input device 412, and the cursor control device 414 may be combined into a single component or device (e.g., an LCD touch screen).

[0085] The data storage device 416 may include a computer-readable storage medium 424 on which is stored one or more sets of instructions 426 that embody any one or more of the methods or functions described herein. During execution of the instructions 426 by the computing device 400, the instructions 426 may also reside completely or at least partially in the main memory 404 and / or the processing device 402, which also constitute computer-readable media. These instructions may also be transmitted or received over the network 418 via the network interface device 422.

[0086] Although the computer-readable storage medium 424 is shown as a single medium in the example embodiment, the term "computer-readable storage medium" may include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "computer-readable storage medium" may also include any medium that is capable of storing, encoding, or carrying a set of instructions executed by a machine and causing the machine to perform any one or more methods of the present disclosure. Therefore, the term "computer-readable storage medium" may include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0087] The terms used in this disclosure and especially in the appended claims (eg, the bodies of the appended claims) are generally considered to be "open terms" (eg, the terms "including" should be interpreted to mean "including, but not limited to").

[0088] Furthermore, if a specific number of claim features are intended to be introduced, such intent will be explicitly stated in the claim, and if there are no such features, such intent does not exist. For example, to aid understanding, the attached claims may contain the introductory phrases "at least one" and "one or more" to introduce claim features. However, even if the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "one" or "a" (for example, "one" and / or "a" should be interpreted as meaning "at least one" or "one or more"), the use of such phrases should not be interpreted as implying that the claim feature introduced by the indefinite article "one" or "a" will limit any particular claim containing such introduced claim feature to an embodiment containing only one such feature. The same is true for the use of definite articles used to introduce claim features.

[0089] In addition, even if the specific number of the technical features introduced in the claims is explicitly stated, those skilled in the art will recognize that such technical features should be interpreted as meaning at least the number of such technical features (for example, a simple statement of "two technical features" without other modifiers means at least two of such technical features, or two or more of such technical features). In addition, when using phrases such as "at least one of A, B, and C" or "one or more of A, B, and C", generally, such a structure is intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.

[0090] In addition, any disjunctive word or phrase preceding two or more alternative terms, whether in the specification, claims or drawings, should be understood to include the possibility of one of the terms, the other of the terms, or both of the terms. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B".

[0091] All examples and conditional language cited in this disclosure are intended for teaching purposes to help readers understand the present disclosure and the concepts contributed by the inventors to advance the art, and should be understood as not being limited to these specifically cited examples and conditions. Although the embodiments of the present disclosure have been described in detail, various changes, substitutions and modifications may be made thereto without departing from the scheme and scope of the present disclosure.

Claims

1. A method comprising: Obtaining, by a data conversion accelerator, a first command and first source data related to the first command; generating, by the data conversion accelerator, one or more containers using the first command, each of the one or more containers having a container size and operable to store compressed source data; Get the operation mode of the data compression operation; performing the data compression operation on a first portion of the first source data using the operation mode to obtain the compressed source data; and The compressed source data is stored in a first container of the one or more containers.

2. The method according to claim 1, wherein: Also included is determining a container size associated with the one or more containers using metadata included in the first command.

3. The method according to claim 1, wherein: The container size is determined based on at least one of an operating system, platform software, storage architecture, performance tuning, and user input.

4. The method according to claim 3, wherein: The user input includes an estimate of the number of containers to be used by each command.

5. The method according to claim 1, further comprising: Obtaining second source data associated with a second command and the first command, wherein the first command and the second command are included in a session; performing the data compression operation on the second portion of the first source data and the first portion of the second source data using the operation mode to obtain second compressed source data; and The second compressed source data is stored in a second container of the one or more containers.

6. The method according to claim 1, wherein: The container sizes of the one or more containers are consistent. 7 . The method of claim 1 , further comprising generating a report associated with the first source data, the report comprising an amount of the first source data consumed in the data compression operation.

8. The method according to claim 1, wherein: The operation mode is stateless data compression, and the data decompression operation on the first container can be independent of the data decompression operation on subsequent containers.

9. The method according to claim 1 also includes reserving a portion of the first container for non-volatile memory standard NVMe protection information (PI) insertion.

10. A data conversion accelerator, comprising: one or more data conversion engines, the one or more data conversion engines including at least a compression engine; and A processing device operable to: obtaining a first command and first source data associated with the first command; generating one or more containers using the first command, each of the one or more containers having a container size and operable to store compressed source data; Get the operation mode of the data compression operation; performing the data compression operation on a first portion of the first source data by the compression engine and using the operation mode to obtain the compressed source data; and The compressed source data is stored in a first container of the one or more containers.

Images