System, method and apparatus for partitioning and encrypting data

Abstract

A method for data encryption may include: receiving input data; finding a delimiter in the input data; generating a data share using a portion of the input data based on a location of the delimiter in the input data; and encrypting the data share. The input data may include records, the delimiter indicates a boundary of a record, and the data share may include records. The location of the delimiter may be in the portion of the input data. Generating the data share may include generating the data share based on a subset of the portion of the input data. The portion of the input data may be a first portion of the input data, and the location of the delimiter may be in a second portion of the input data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefits of U.S. Provisional Patent Application Serial No. 63 / 231,709, entitled “Object Processing and Filtering for Computational Storage,” filed on August 10, 2021, U.S. Provisional Patent Application Serial No. 63 / 231,711, entitled “Data Placement with Spatial Locality and Hierarchical Aggregation for Computational Storage,” filed on August 10, 2021, U.S. Provisional Patent Application Serial No. 63 / 231,710, entitled “Integrated Data Chunking and Compression for Near Data Processing,” filed on August 10, 2021, and U.S. Provisional Patent Application Serial No. 63 / 231,715, entitled “Integrated Data Chunking and Encryption for Near Data Processing,” filed on August 10, 2021, which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to computing storage systems and, more particularly, to systems, methods, and apparatus for partitioning and encrypting data. Background Art

[0004] A computational storage device may include one or more processing resources that may perform operations on data stored on the device. A host may offload processing tasks to a storage device, for example, by sending a command to the storage device indicating an operation to be performed on data stored on the device. The storage device may use one or more processing resources to execute the command. The storage device may send the results of the operation to the host and / or store the results on the device.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the principles of the invention and therefore it may contain information that does not constitute the prior art. Summary of the invention

[0006] A method for data compression may include receiving input data, searching for a delimiter in the input data, generating data shares using portions of the input data based on a location of the delimiter in the input data, and compressing the data shares. The input data may include records, the delimiter indicates a boundary of a record, and the data shares may include the record. The location of the delimiter may be in the portion of the input data. Generating the data shares may include generating the data shares based on a subset of the portion of the input data. The portion of the input data may be a first portion of the input data, and the location of the delimiter may be in a second portion of the input data. Generating the data shares may include generating the data shares based on the first portion of the input data and the second portion of the input data. The size of the portion of the input data may be based on a default share size. The method may also include modifying the size of the data shares based on the location of the delimiter. Modifying the size of the data shares may include expanding the size of the data shares. Receiving may include receiving an input data stream. Searching for the delimiter in the input data may include performing a first scan operation on the input data, and compressing the data shares may include performing a second scan operation on the data shares.

[0007] A method for data compression may include scanning input data, performing a compression operation based on the scanning to generate compressed data using the input data, finding a delimiter in the input data based on the scanning, and generating data shares using the compressed data based on the position of the delimiter in the input data. The input data may include records, the delimiter indicates a boundary of the record, and the data share may include records. The generating may include generating the data share based on a share size. The share size may be a default share size. The share size may be based on the default share size and the length of a match in the input data. The match may include a delimiter. The delimiter may be a first delimiter, the method may further include maintaining the first delimiter and finding a second delimiter in the input data, wherein the generating may include generating the data share based on the first delimiter and the second delimiter. The method may further include setting an indication based on the size of the compressed data. The indication may include a termination indication. The generating may include generating the data share based on the indication and the delimiter. The scanning may include scanning the input data based on the indication. The scanning may include scanning the input data based on the delimiter. The performing may include performing a compression operation based on the delimiter. The input data may include an input data stream. The compression operation may include a stream-based compression operation.

[0008] A system may include a host, the host including host logic, the host logic configured to perform a scan operation on input data, perform a data compression operation based on the scan operation to generate compressed data using the input data, find a delimiter in the input data based on the scan operation, and generate a data share using the compressed data based on the delimiter. The host logic may also be configured to generate the data share based on a default share size and a position of the delimiter in the input data. The delimiter may be a first delimiter, and the host logic may also be configured to generate the data share based on a position of a second delimiter in the input data. The system may also include a device configured to receive the data share from the host, the device including device logic configured to decompress the data share to generate a decompressed data share, and perform an operation on the decompressed data share.

[0009] A method for data encryption may include receiving input data; searching for a delimiter in the input data; generating a data share using a portion of the input data based on a location of the delimiter in the input data; and encrypting the data share. The input data may include records, the delimiter indicates a boundary of a record, and the data share may include records. The location of the delimiter may be in the portion of the input data. Generating the data share may include generating the data share based on a subset of the portion of the input data. The portion of the input data may be a first portion of the input data, and the location of the delimiter may be in a second portion of the input data. Generating the data share may include generating the data share based on the first portion of the input data and the second portion of the input data. The size of the portion of the input data may be based on a default share size. The method may also include modifying the size of the data share based on the location of the delimiter. Modifying the size of the data share may include extending the size of the data share. Receiving the input data may include receiving an input data stream. Searching for the delimiter in the input data may include performing a first scan operation on the input data, and encrypting the data share may include performing a second scan operation on the data share.

[0010] A method for data encryption may include: scanning input data; based on the scanning, performing an encryption operation to generate encrypted data using the input data; based on the scanning, finding a delimiter in the input data; and generating a data share using the encrypted data based on the position of the delimiter in the input data. The input data may include records, the delimiter indicates a boundary of the record, and the data share may include records. The generating may include generating the data share based on a share size. The share size may be a default share size. The generating may include extending the size of the data share based on the position of the delimiter in the input data. The encryption operation may include a block-based encryption operation, and the performing may include performing the encryption operation on a first block of the input data to generate a first block of encrypted data. The delimiter may be located in a second block of the input data, and the method further includes extending the size of the data share based on the position of the delimiter. The method may also include extending the size of the data share based on the size of the second block of the input data. The extending may include padding the data share based on the size of the second block of the input data. The size of the second block of the input data may be based on a key length of the encryption operation. The performing may also include performing the encryption operation on the second block of the input data to generate a second block of encrypted data. The generating may include generating the data share using the first block of encrypted data and the second block of encrypted data.

[0011] A system may include a host, the host including host logic, the host logic configured to perform a scan operation on input data, perform a data encryption operation based on the scan operation to generate encrypted data using the input data, find a delimiter in the input data based on the scan operation, and generate a data share using the encrypted data based on the delimiter. The host logic may also be configured to generate the data share based on a default share size and a position of the delimiter in the input data. The host logic may also be configured to generate the data share based on a key length of the encryption operation. The system may also include a device configured to receive the data share from the host, the device including device logic configured to decrypt the data share to generate a decrypted data share and perform an operation on the decrypted data share.

[0012] A method of partitioning data may include: scanning input data; based on the scan, performing an operation using the input data to generate processed data; based on the scan, finding a delimiter in the input data; and generating a data share using the processed data based on the position of the delimiter in the input data. The operation may include a data compression operation. The operation may include a data encryption operation. The generating may include generating the data share based on a default share size. The input data may include an input data stream, and the operation may include a stream-based operation. The operation may include a block-based operation. The performing may include performing the operation using a block of the input data. The generating may include modifying the data share based on the position of the delimiter in the input data. The modifying may include extending the data share. The extending may include extending the data share based on a block size of the operation. The extending may include padding the data share.

[0013] A system may include a host, the host including host logic, the host logic configured to perform a scan operation on input data, perform a processing operation based on the scan operation to generate processed data using the input data, find a delimiter in the input data based on the scan operation, and generate a data share using the processed data based on the delimiter. The host logic may also be configured to generate the data share based on a default share size and a position of the delimiter in the input data. The system may also include a device configured to receive the data share from the host, the device including device logic configured to recover the data share to generate a recovered data share and perform an operation on the recovered data share. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings are not necessarily drawn to scale, and in all drawings, for the purpose of illustration, elements of similar structure or function may generally be represented by similar reference numerals or portions thereof. The drawings are only for the convenience of describing the various embodiments described herein. The drawings do not describe every aspect of the teachings disclosed herein and do not limit the scope of the claims. In order to prevent the drawings from becoming obscure, not all components, connections, etc. are shown, and not all components have reference numerals. However, the mode of component configuration can be easily seen from the drawings. The drawings, together with the specification, illustrate example embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure.

[0015] Figure 1A An embodiment of an object storage solution using server-side encryption according to an exemplary embodiment of the present disclosure is shown.

[0016] Figure 1B An embodiment of an object storage solution using client-side encryption according to an exemplary embodiment of the present disclosure is shown.

[0017] Figure 2AAn embodiment of an object storage solution that can return objects to a user device according to an exemplary embodiment of the present disclosure is shown.

[0018] Figure 2B An embodiment of an object storage solution with a data selection feature according to an exemplary embodiment of the present disclosure is shown.

[0019] Figure 3A An embodiment of a write operation of an object storage solution with a data selection feature according to an exemplary embodiment of the present disclosure is shown.

[0020] Figure 3B An embodiment of a read operation of an object storage solution with a data selection feature according to an exemplary embodiment of the present disclosure is shown.

[0021] Figure 4 An embodiment of a storage system with local data recovery according to an example embodiment of the present disclosure is shown.

[0022] Figure 5 Another embodiment of a storage system with local data recovery according to an exemplary embodiment of the present disclosure is shown.

[0023] Fig. 6A An example embodiment of a write operation of a storage solution with local data recovery and server-side encryption according to an example embodiment of the present disclosure is shown.

[0024] Figure 6B An example embodiment of a write operation of a storage solution with local data recovery and client-side encryption according to an example embodiment of the present disclosure is shown.

[0025] Fig. 7A An example embodiment of a write operation of a storage scheme with local data recovery according to an example embodiment of the present disclosure is shown.

[0026] Figure 7B An example embodiment of a read operation with data selection of a storage scheme with local data recovery according to an example embodiment of the present disclosure is shown.

[0027] Figure 8 An exemplary embodiment of a system architecture of an object storage solution with local data recovery according to an exemplary embodiment of the present disclosure is shown.

[0028] Fig. 9A An example embodiment of a read and / or write operation of a storage scheme with local data recovery according to an example embodiment of the present disclosure is shown.

[0029] Fig. 9B An example embodiment of a read operation and a data selection operation of a storage scheme with local data recovery according to an example embodiment of the present disclosure is shown.

[0030] Fig.10 An embodiment of distribution of data from Table 1 on three data partitions at three computing storage devices according to an example embodiment of the present disclosure is shown.

[0031] Fig.11 An example embodiment of a storage system according to an example embodiment of the present disclosure is shown, in which a server can reconstruct records split between data chunks at different storage devices.

[0032] Fig.12 An embodiment of a content-aware method for dividing data into blocks and compressing the data according to an exemplary embodiment of the present disclosure is illustrated.

[0033] Fig.13 An embodiment of a method for compressing data according to an exemplary embodiment of the present disclosure is shown.

[0034] Fig.14 Show that it can be Fig.13 The illustrated method is an embodiment of the operations performed.

[0035] Fig.15 An embodiment of an integrated partitioning and compression scheme according to an example embodiment of the present disclosure is shown.

[0036] Fig.16 It is shown that according to an exemplary embodiment of the present disclosure, Fig.15 The illustrated scheme implements an embodiment of an integrated segmentation and compression method.

[0037] Fig.17 Another embodiment of an integrated segmentation and compression scheme according to an example embodiment of the present disclosure is shown.

[0038] Fig.18 It is shown that according to an exemplary embodiment of the present disclosure, Fig.17 The illustrated scheme implements an embodiment of an integrated segmentation and compression method.

[0039] Fig.19 Another embodiment of the integrated blocking and compression method according to an example embodiment of the present disclosure is shown.

[0040] Fig. 20 An embodiment of a system with integrated chunking and compression according to an example embodiment of the present disclosure is shown.

[0041] Fig.21 An embodiment of a content-aware method for dividing data into blocks and encrypting the data according to an exemplary embodiment of the present disclosure is illustrated.

[0042] Fig. 22 An example embodiment of a block cipher method that can be used for encryption data technology according to an example embodiment of the present disclosure is shown.

[0043] Fig.23 An embodiment of an integrated data segmentation and encryption scheme according to an example embodiment of the present disclosure is shown.

[0044] Fig.24 An embodiment of an integrated data segmentation and encryption method according to an exemplary embodiment of the present disclosure is shown.

[0045] Fig.25 An embodiment of a system with integrated segmentation and encryption according to an example embodiment of the present disclosure is shown.

[0046] Fig.26 An example embodiment of a host device for a storage scheme with data block modification according to an example embodiment of the present disclosure is shown.

[0047] Fig. 27 An example embodiment of a storage device with data block modification according to an example embodiment of the present disclosure is shown.

[0048] Fig.28 An embodiment of a method for data compression according to an exemplary embodiment of the present disclosure is shown.

[0049] Fig.29 An embodiment of a method for data encryption according to an exemplary embodiment of the present disclosure is shown.

[0050] Fig.30 An embodiment of a method for chunking data according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0051] The object storage system may implement a data selection feature that enables a user's device to request retrieval of a specified subset of data from a stored object. To process such a request, the storage server may reconstruct the object from one or more data portions stored on one or more storage devices. The storage server may also decrypt the object if it is encrypted, and / or decompress the object if it is compressed, to restore the object to its original form. The storage server may perform one or more selection operations on the restored object, such as filtering, scanning, etc., to find the specified subset of data requested by the user's device. The storage server may return the requested subset of data to the user's device.

[0052] In some aspects, a computing storage device is capable of performing one or more selection operations, such as filtering, scanning, etc., on objects stored on the device. However, if shares of an object are stored on the device, and the object is modified (e.g., compressed, encrypted, etc.) prior to partitioning the data into shares, the shares stored on the device may include only random (to the device) information that the storage device may not be able to recover (e.g., decompress and / or decrypt) into the original data. As a result, the storage device may not be able to locally perform meaningful operations on the shares of data stored on the device.

[0053] The present disclosure includes many principles related to computational storage. The principles disclosed herein may have independent utility and may be implemented separately, and not every embodiment may utilize every principle. In addition, these principles may also be embodied in various combinations, some of which may amplify some benefits of a single principle in a synergistic manner.

[0054] Some principles disclosed herein relate to dividing data into one or more shares before performing one or more modifications to the one or more shares. For example, in a computational storage scheme according to an example embodiment of the present disclosure, an object or other original data may be divided into data shares before performing modifications such as compression and / or encryption on the data. One or more data shares may be modified individually (e.g., compression and / or encryption may be performed on individual shares of the data), and modified versions of the data shares may be sent to a computational storage device for storage and / or processing. The storage device may generate a restored version of the data share from the modified data share, for example by decrypting and / or decompressing the modified data share. The storage device may perform operations (e.g., selection operations) locally on the restored data shares.

[0055] Depending on implementation details, performing the selection operation locally on the computing storage device can reduce the amount of data sent from one or more storage devices to the server. In addition, depending on implementation details, the computing storage device can perform operations such as the selection operation more efficiently than the server.

[0056] In some example embodiments according to the present disclosure, a storage device, a storage server, or the like may provide one or more instructions on how to divide the original data into multiple shares and / or how to modify the shares to facilitate storage and / or processing by one or more computing storage devices. For example, in some embodiments, the instructions may include information such as one or more share sizes, compression algorithms, encryption algorithms, etc. that the storage device may support. In some embodiments, one or more instructions may be mandatory, optional (e.g., advisory), or a combination thereof. For example, an indication of an optimal share size for storage on a particular storage device may be advisory, while an indication of supported compression algorithms may be mandatory to enable the storage device to decompress the data shares for local processing on the device.

[0057] According to example embodiments of the present disclosure, any operations disclosed herein, including dividing data, modifying data (e.g., compressing and / or encrypting data), erasure coding data, storing data, processing data, selecting data, etc., can be distributed (e.g., mapped) in various devices in unlimited configurations. For example, in some embodiments, a client may divide original data (e.g., an object) into one or more shares, compress the data shares, and send the compressed data shares to a server. The server may encrypt the compressed data shares and store the compressed and encrypted data shares on one or more storage devices. As another example, in some embodiments, a client may divide original data (e.g., an object) into one or more shares, compress and encrypt the data shares, and send the compressed and encrypted data shares to a server for storage on one or more storage devices. As another example, a client may send original data (e.g., an object) to a server, and the server may divide the data into one or more shares, and compress, encrypt, and / or perform erasure coding on the data shares, and store the individually modified data shares across one or more storage devices.

[0058] Some additional principles of the present disclosure relate to content-aware techniques for dividing data into multiple shares. For example, in some embodiments, the size of a share can be dynamically determined by analyzing the content of the share to find the boundaries of records within the divided data (e.g., indicated by delimiters). The size of a share can be determined to align with one or more complete records in the share. For example, if a data share with a default share size includes incomplete records, the size of the share can be modified (e.g., expanded or reduced) so that the share ends with a delimiter of records within the share (e.g., the share may only include complete records). The resulting self-contained share can be compressed and / or encrypted as a unit.

[0059] Some additional content-aware data partitioning techniques according to example embodiments of the present disclosure may integrate a data partitioning operation with another operation that may scan the data to be partitioned. For example, in some embodiments, a data partitioning operation may be combined with a data compression operation so that when an input data stream is scanned for compression purposes, it may also be scanned for one or more delimiters indicating one or more boundaries of one or more records. The data share size may be determined by the location of the one or more delimiters so that the share ends at the end of the record. In some embodiments, the data compression operation may also terminate at the end of the share. Depending on implementation details, this may improve the efficiency of the data partitioning operation because it may take advantage of data scans that have already been performed for compression purposes.

[0060] As another example, in some embodiments, the data partitioning operation can be combined with the data encryption operation. When the input data stream is read for encryption purposes, it can also be scanned to find one or more delimiters indicating one or more boundaries of one or more records. The size of the data share can be determined by the position of the one or more delimiters, so that the share ends at the end of the record. In some embodiments, the data encryption operation can also terminate at the end of the share. If the encryption operation is implemented using a block-cipher algorithm, the size of the share can be modified (e.g., extended or reduced) so that the end of the share is aligned with the end of the block (if the size of the data is not an even multiple of the block size, padding can be performed).

[0061] In some embodiments, a portion of data may also be referred to as a chunk of data, and dividing data into a plurality of portions of data or chunks of data may be referred to as chunking the data. In some embodiments, a portion of data or chunk of data may refer to any unit of data that may be obtained by, for example, dividing data for the purpose of storage on one or more storage devices. In some cases, if the amount of original data is less than or equal to a portion or chunk size (e.g., a default portion or chunk size), the unit of original data generated by the division or chunking operation may still be referred to as a portion of data or chunk, even if its size is the same as the amount of original data.

[0062] For purposes of illustration, some embodiments may be described in the context of an object storage system that may implement data selection features and / or may store data in one or more key-value (KV) storage devices. However, the principles described in the present disclosure are not limited to any particular data format, data processing features, storage device interfaces, etc. For example, the systems, methods, and / or apparatus according to the example embodiments of the present disclosure may also be implemented with a storage system that may provide file storage, database storage, block storage, etc., may implement any type of processing features such as acceleration, chart processing, graphics processing, machine learning, etc., and may operate with any type of storage device including KV storage devices, block storage devices, etc.

[0063] An object storage system may enable a user's device to store data in the form of objects. The data in an object may be modified in various ways before being stored. For example, data may be compressed to reduce the amount of space it occupies in a storage medium and / or to reduce the time, bandwidth, power, etc. required to transfer the data from a client to one or more storage devices (e.g., over a network). As another example, the data in an object may be encrypted to prevent unauthorized access to the data during transmission and / or storage of the data.

[0064] An object may include a relatively large amount of data, and therefore, for purposes of reliability, accessibility, etc., the object may be divided into blocks that may be stored across multiple storage devices. (Dividing data into blocks may also be referred to as chunking the data.) For example, after compression and / or encryption, the object may be divided into fixed-size blocks to fit the block size used by one or more block-based storage devices in the storage system. In some embodiments, an erasure coding scheme may be used to divide the data into data chunks and generate one or more parity chunks that may enable the storage system to recover lost or damaged data chunks.

[0065] Figure 1A An embodiment of an object storage scheme with server-side encryption according to an example embodiment of the present disclosure is shown. Figure 1A The left side shows the data flow between system components during read and / or write operations. Figure 1A The right side of shows the manipulation of data during a write operation.

[0066] Figure 1A The system shown on the left may include a client 102, one or more servers 104 (collectively referred to as servers), and one or more storage devices 108 (collectively referred to as storage). Figure 1A The operations shown on the right side of are shown in a first group 110A performed by the client 102 and a second group 112A performed by the server 104 .

[0067] During a write operation, the client 102 may start with raw data 114, which may be, for example, an object. The client 102 may perform one or more compression operations on the raw data 114 to generate compressed data 116. The client 102 may send the compressed data 116 to the server 104, which may encrypt the compressed data 116 to generate encrypted data 118. The server 104 may divide the compressed and encrypted data 118 into one or more data chunks 120 and send the one or more data chunks 120 to the one or more storage devices 108. In some embodiments, the server 104 may perform erasure coding on the one or more data chunks 120 to generate one or more parity chunks 121 that may also be stored on the one or more storage devices 108.

[0068] During a read operation, Figure 1A The operations shown in can be performed in reverse. For example, the server 104 can read one or more data chunks 120 from the one or more storage devices 108. If one of the data chunks is lost or damaged, for example due to a failed storage device, the server 104 can use one or more parity chunks 121 to recover the lost or damaged data chunk. The server 104 can reconstruct the compressed and encrypted data 118 from the data chunks 120. The server 104 can decrypt the compressed and encrypted data 118 and send the compressed and decrypted data 116 to the client 102. The client 102 can decompress the compressed and decrypted data 116 to recover the original data 114, which can be, for example, an object.

[0069] Figure 1B An embodiment of an object storage scheme with client-side encryption according to an example embodiment of the present disclosure is shown. Figure 1B The left side shows the data flow between system components during read and / or write operations. Figure 1B The right side of shows the manipulation of data during a write operation.

[0070] exist Figure 1B The system shown on the left and Figure 1B The operations shown on the right side may include Figure 1A Some components and / or operations are similar to those shown in FIG. 1 and may be denoted by the same or similar reference numerals. Figure 1BIn the illustrated embodiment, the client 102 may encrypt the compressed data 116 to generate compressed and encrypted data 118, as shown by a first set of operations 110B performed by the client 102. The client 102 may send the compressed and encrypted data 118 to the server 104, and the server 104 may divide the compressed and encrypted data 118 into one or more data chunks 120, as shown by a second set of operations 112B performed by the server 104. The server 104 may send the one or more data chunks 120 to the one or more storage devices 108. In some embodiments, the server 104 may perform erasure coding on the one or more data chunks 120 to generate one or more parity chunks 121 that may also be stored on the one or more storage devices 108.

[0071] During a read operation, Figure 1B The operations shown in 120 may be performed in reverse. For example, the server 104 may reconstruct the compressed and encrypted data 118 from the data chunks 120 (using one or more parity chunks 121 to recover any lost or damaged data chunks, if necessary), and send the compressed and encrypted data 118 to the client 102. The client 102 may decrypt the compressed and encrypted data 118 to generate the compressed and decrypted data 116. The client 102 may decompress the compressed and decrypted data 116 to recover the original data 114, which may be, for example, an object.

[0072] Figure 1A and Figure 1B The illustrated embodiments are exemplary embodiments only, and the number, order, and / or arrangement of components and / or operations may vary. For example, in some embodiments, the original data 114 may be stored without compression and / or encryption. In some embodiments, one or more servers 104 may be implemented with a first server that may be configured as an object storage server and a second server that may be configured as a storage server (also referred to as a storage node) to manage one or more storage devices 108. Thus, the first and second servers may implement an object storage service. If any or all of the original data 114 is encrypted, the encryption key may be generated by the storage service and / or user of the service. In some embodiments, performing a chunking operation at or near the end of a write operation may enable the server 104 to divide the data into chunks, the size of which may correspond to one or more block sizes of one or more storage devices 108.

[0073] In some cases, a user associated with a user device may need to retrieve only a subset of the data stored in an object. Some object storage systems may require the user to retrieve the entire object and process the object to find the subset of data. This may result in a relatively large amount of unnecessary data being transferred to the user's device, which in turn may consume unnecessary resources, such as time, bandwidth, power, etc.

[0074] To reduce and / or prevent the transmission of unwanted data, some object storage systems may provide a data selection feature that enables a user to request retrieval of a specified subset of data from a stored object. Instead of sending the entire object to the user's device, the object storage system may perform scanning, filtering, and / or other data selection operations on the object to find the specified subset of data. The object storage system may return the specified subset of data to the user's device.

[0075] Figure 2A An embodiment of an object storage solution that can return objects to a user device according to an example embodiment of the present disclosure is shown. Figure 2B An embodiment of an object storage solution with a data selection feature according to an example embodiment of the present disclosure is shown.

[0076] refer to Figure 2A , object storage service 201 can store objects 203A, 203B, and 203C for the user in a data bucket or container 205. If the user needs to retrieve a data subset (e.g., one or more records) from one of objects 203A, object storage service 201 can require the user to request the entire object 203A, which can be sent to client computing operation 207 over the network. Client computing operation 207 can perform data selection operation 209 on object 203A, such as scanning, filtering, etc., to find the data subset. Client computing operation 207 can use the data subset for further operations 211.

[0077] refer to Figure 2B , an object storage service 213 with a data selection feature may enable a user to request a subset of data from a stored object 203A. For example, the object storage service 213 may enable a user to submit a request, for example, by sending a query (e.g., using an expression in a database language such as SQL), which may operate on objects 203A that may be stored, for example, in a format such as comma separated variables (CSV), JavaScript object notation (JSON), Parquet, etc. In some embodiments, the query may be sent to the object storage service 213, for example, using an application programming interface (API), a software development kit (SDK), etc.

[0078] Instead of sending the entire object 203A, the object storage service 213 may perform data selection operations 209, such as scanning, filtering, etc., on the object 203A to find the data subset specified by the user in the request. The object storage service 213 may send the data subset 213a to the client computing operation 217 for further operation 211. Depending on the implementation details, the object storage service 213 may perform one or more recovery operations 219, such as decompression, decryption, etc., on the object 203A to reverse the compression operation, encryption operation, etc. that may have been performed on the object 203a when the object 203A was stored.

[0079] Figure 3A An embodiment of a write operation of an object storage solution with a data selection feature according to an example embodiment of the present disclosure is shown. Figure 3B An embodiment of a read operation of an object storage solution with a data selection feature according to an exemplary embodiment of the present disclosure is shown. For example, Figure 3A and Figure 3B The illustrated embodiment can be used to implement Figure 2B The object storage scheme shown.

[0080] Figure 3A The left side shows the data flow between components of the object storage system during read and / or write operations. Figure 3A The right side of shows the manipulation of data during a write operation.

[0081] Figure 3A The system shown on the left may include a client 302 , one or more servers 304 (which may be collectively referred to as servers), and one or more storage devices 308 (which may be collectively referred to as storage). Figure 3A The operations shown on the right are shown in a first set 310A performed by the client 302 and a second set 312A performed by the server 304 . Figure 3A The data flow between the components and / or data operations shown in can be similar to Figure 1A or the embodiment with server-side encryption shown in Figure 1B , where elements with reference numbers ending with the same number may be similar. Figure 3A , compressed and encrypted data 318 may be part of group 310A for an implementation of client-side encryption, or part of group 312A for an implementation of server-side encryption.

[0082] refer to Figure 3B, a user may request a data subset from an object or other raw data stored on one or more storage devices 308. To process such a request, server 304 may read one or more data chunks 320 from one or more storage devices 308. If one of the data chunks is lost or damaged, server 304 may use one or more parity chunks 321 to recover the lost or damaged data chunk. Server 304 may reconstruct compressed and encrypted data 318 from data chunks 320.

[0083] The server 304 may decrypt the compressed and encrypted data 318 to generate compressed and decrypted data 316, which may be decompressed to recover the original data 314 (e.g., object). The server 304 may perform a data selection operation (e.g., scan, filter, etc.) on the original data 314 to obtain the requested data subset 323. The server 304 may send the data subset 323 to the client 302. Because the client's decompression operation may be bypassed, it is grayed out. Figure 3B The operations shown on the right side of are shown in group 312B as being performed by server 304.

[0084] as Figure 1A and Figure 1B As in the embodiment shown, Figure 3A and Figure 3B The illustrated server 304 may be implemented with a first server that may be configured as an object storage server and a second server that may be configured as a storage server to manage one or more storage devices 308. Thus, in some embodiments, the storage server may reconstruct the compressed and encrypted data 318 from the one or more data chunks 320, and the object storage server may perform decryption, decompression, and / or data selection operations. In addition, although Figure 3A and Figure 3B The illustrated embodiment may implement server-side encryption, but other embodiments may also implement client-side encryption.

[0085] Depending on the implementation details, Figure 3A and Figure 3B The embodiments shown in can reduce network traffic, for example, by reducing the amount of data transferred between the storage system and the client. Figure 3A and Figure 3BThe data processing flow of the illustrated architecture may prevent the storage system from utilizing a computational storage device, which, depending on implementation details, may be well suited to perform some or all of the operations performed by the server 304. For example, in some embodiments, the computational storage device may include processing resources that can perform decompression, decryption, and / or other operations such as data selection operations more efficiently than general-purpose processing resources that may be present in the server. However, because the original data 314 may be modified (e.g., compressed, encrypted, etc.) prior to chunking, the separate storage device 308 storing the data chunks may not be able to decrypt, decompress, and / or otherwise restore the data chunks to a form in which meaningful operations can be performed locally on the device.

[0086] Figure 4 An embodiment of a storage system with local data recovery according to an example embodiment of the present disclosure is shown. Figure 4 The system shown may include a host 424 and a computing storage device 408, which may communicate via a connection 422. The host 424 may include data chunking logic 426 (which may also be referred to as data partitioning logic) and data modification logic 427, which may be configured to provide one or more data chunks to the storage device 408 in a form in which the storage device 408 can restore the data chunks to a form in which the storage device can perform operations on them. For example, the data chunking logic 426 may divide an object or other raw data into one or more data chunks before being modified by the data modification logic 427. The data chunking logic 426 may or may not be content-aware. For example, in an embodiment in which the data chunking logic 426 is not content-aware, the data chunking logic 426 may divide the data into fixed-size chunks, so that one or more chunks may include one or more record fragments. As another example, in an embodiment where the data chunking logic 426 is content-aware, the data chunking logic 426 may look for record delimiters in the data to divide the data into chunks of variable size such that a chunk may end with a record delimiter and therefore a complete record.

[0087] Data modification logic 427 may perform one or more data modification operations, such as compression, encryption, erasure coding, etc., on one or more blocks individually to generate one or more modified blocks of original data. Host 424 may send the one or more modified blocks of original data to computational storage device 408 and / or one or more additional computational storage devices for storage and / or processing.

[0088] The computational storage device 408 may include data recovery logic 428, one or more processing elements 429, and storage media 430. The data recovery logic 428 may be configured to recover the modified data blocks into a form on which the one or more processing elements 429 can perform operations. For example, the data recovery logic 428 may decrypt encrypted modified data blocks, decompress compressed modified data blocks, and so on. The one or more processing elements 429 may be configured to perform any type of operation, such as data selection (e.g., scanning, filtering, etc.), computational acceleration, chart processing, graphics processing, machine learning, and so on. The storage medium 430 may be used to store any data, including the one or more modified data blocks sent by the host 424.

[0089] In some embodiments, the data recovery logic 428 and / or one or more processing elements 429 may be configured to read and recover one or more data chunks from the storage medium 430 in response to a request received at the storage device 408 that may include a query (e.g., an expression) and return a specified subset of data, or perform any other operation on the recovered data chunks.

[0090] In some embodiments, the restored data chunks may or may not be identical to the original data before chunking. For example, if the data chunks stored at storage device 424 contain financial information such as bank account transactions, balances, etc., and the user only requests the account balance, the recovery logic 428 and / or one or more processing elements 429 may need to restore the data chunks to their original form to find the accurate account balance and send them to the user's device. However, if the data chunks stored at storage device 424 contain photo images, and the user requests a list of features in the image, the recovery logic 428 and / or one or more processing elements 429 may only need to decompress the image to the extent that one or more processing elements 429 can recognize the features requested by the user.

[0091] The host 424 may be implemented with any component or combination of components that can provide one or more data chunks to the storage device 408 in a form that the storage device 408 can recover and / or perform operations on. For example, in some embodiments, the host 424 may include a client, an object storage server, and / or a storage node. The data chunking logic 426 and / or the data modification logic 427 may be distributed between any components of the host 424 in any manner. For example, in some embodiments, the data chunking logic 426 may be implemented on the client, while the data modification logic 427 may be implemented on the object storage server and / or the storage node. As another example, the data chunking logic 426 and the data modification logic 427 including the compression logic may be implemented on the client, while the data modification logic 427 including the encryption and / or erasure coding logic may be implemented on the server. Thus, the client may divide the original data into chunks, compress the data chunks individually, and send the compressed data chunks to the server. The server may individually encrypt the compressed data chunks, perform erasure coding on the data chunks to generate one or more parity chunks, and store the data chunks and / or parity chunks on one or more storage devices including the computational storage device 408 .

[0092] The storage device 408 and / or any other storage device disclosed herein may be implemented in any form factor (such as 3.5-inch, 2.5-inch, 1.8-inch, M.2, enterprise and data center SSD form factor (EDSFF), NF1, etc.) using any connector configuration such as Serial ATA (SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), M.2, U.2, U.3, etc.

[0093] The storage device 408 and / or any other storage device disclosed herein may be implemented with any storage medium 430, including solid-state media, magnetic media, optical media, etc., or any combination thereof. Examples of solid-state media may include flash memory (such as NAND flash memory, low-latency NAND flash memory), persistent memory (PMEM) (such as cross-grid non-volatile memory, memory with body resistance change, phase change memory (PCM), etc.), or any combination thereof.

[0094] The storage device 408 and / or any other storage device disclosed herein may communicate using any type of storage interface and / or protocol, such as Peripheral Component Interconnect Express (PCIe), NVMe, NVMe-over-fabric (NVMe-oF), NVMe Key Value (NVMe-KV), SATA, SCSI, etc., or any combination thereof. In some embodiments, the storage device 408 and / or any other storage device disclosed herein may implement a coherent (e.g., memory coherent, cache coherent, etc.) or memory semantic interface, such as Compute Express Link (CXL), and / or a coherent protocol, such as CXL.mem, CXL.cache, and / or CXL.IO. Other examples of coherent and / or memory semantic interfaces and / or protocols may include Gen-Z, a coherent accelerator processor interface (CAPI), a cache coherent interconnect for accelerators (CCIX), etc.

[0095] The storage device 408 and / or any other storage device disclosed herein, and any components of the host 424 (e.g., a client, an object storage server, a storage node, etc.) can be implemented in whole or in part with a server chassis, a server rack, a data room, a data center, an edge data center, a mobile edge data center, and / or any combination thereof, and / or used in conjunction with them.

[0096] The communication connection 422 and / or any other connection disclosed herein, including any connection between components such as clients, servers, storage devices, etc., can be implemented with any interconnect and / or network interface and / or protocol, including PCIe, Ethernet, Transmission Control Protocol / Internet Protocol (TCP / IP), Remote Direct Memory Access (RDMA), RDMA over Converged Ethernet (ROCE), Fibre Channel, InfiniBand, iWARP, etc., and / or any combination thereof.

[0097] Any functionality disclosed herein, including any logic such as data partitioning logic 426, data modification logic 427, data recovery logic 428, one or more processing elements 429, indication logic 531, etc., can be implemented in hardware, software, or a combination thereof, including combinational logic, sequential logic, one or more timers, counters, registers, and / or state machines, one or more complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), central processing units (CPUs) (such as complex instruction set computers (CISC) processors such as x86 processors and / or reduced instruction set computers (RISC) processors such as ARM processors), graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), etc., executing instructions stored in any type of memory, or any combination thereof. In some embodiments, one or more of the data recovery logic 428, processing elements 429, etc. may include fixed and / or programmable functions to perform any function, such as compression and / or decompression, encryption and / or decryption, microservices, erasure coding, video encoding and / or decoding, database acceleration, search, machine learning, graph processing, etc. In some embodiments, one or more components may be implemented as a system on a chip (SOC).

[0098] In some embodiments, one or more of the data recovery logic 428, the processing element 429, etc. may be integrated with one or more other components of the storage device, such as a storage device controller, a flash translation layer (FTL), etc.

[0099] Any data modification operations disclosed herein, such as compression, encryption, etc. (or their inverse operations), can be implemented using any suitable technology. For example, data compression and / or decompression can be implemented using LZ77, gzip, Snappy, etc. Encryption and / or decryption can be implemented using Advanced Encryption Standards (AES) such as AES-256, Rivest-Shamir-Adleman (RSA), etc.

[0100] Figure 5 Another embodiment of a storage system with local data recovery according to an example embodiment of the present disclosure is shown. Figure 5 The system shown may include a Figure 4 The components and / or implementations described in the illustrated embodiments are similar, and elements having reference numerals ending with the same number may be similar. Figure 5 In the illustrated embodiment, the computational storage device 508 may also include indication logic 531 , which may be configured to provide one or more indications 532 to the data blocking logic 526 and / or the data modification logic 527 at the host 524 .

[0101] The one or more indications 532 may include information that may be used by the data chunking logic 526 to determine how to divide the raw data into chunks. For example, the one or more indications 532 may include one or more storage hyperparameters such as a minimum chunk size, a maximum chunk size, an optimal chunk size, etc. for storage utilization, processing efficiency (e.g., chunk decompression, decryption, data selection, and / or other operations), bandwidth utilization, etc.

[0102] The one or more indications 532 (e.g., storage hyperparameters) may include information that may be used by the data modification logic 527 to determine how to modify the individual data chunks provided by the data chunking logic 526. For example, the one or more indications 532 may include a list of types of compression algorithms, encryption algorithms, etc. supported by the data recovery logic 528 at the storage device 508.

[0103] In some embodiments, one or more indications may be mandatory, optional (e.g., provided as suggestions), or a combination thereof. For example, an indication of an optimal chunk size for storage on storage device 508 may be provided as a suggestion, while an indication of one or more compression algorithms, encryption algorithms, etc. supported by data recovery logic 528 may be mandatory to enable storage device 508 to decompress and / or decrypt data chunks for local processing by one or more processing elements 529 at storage device 508.

[0104] In some embodiments, indication logic 531 may be located entirely within computing storage device 508. However, in some other embodiments, indication logic 531 may be located at host 524, distributed between host 524 and storage device 508 or multiple storage devices, or located entirely at a different device (e.g., a separate server, management controller, etc. that may maintain a list or database of characteristics of storage devices in the system). For example, in some embodiments, one or more storage nodes may include indication logic 531, which may maintain a list or database of indications of each storage device installed on the storage node and provide the indications to one or more clients, object storage servers, etc. As another example, one or more storage nodes may include a portion of indication logic 531 that may maintain an indication of each storage device installed on the storage node, and an object storage server may include a portion of indication logic 531 that may aggregate indications from one or more storage nodes and provide the indications to one or more clients.

[0105] For example, in response to a query, command, etc. (e.g., an NVMe command, a query through an API, an SDK, etc.), the indication logic 531 may provide any indication 532 to any device, such as a client, an object storage server, a storage node, etc. In some embodiments, one or more indications 532 (e.g., one or more storage hyperparameters) may be provided to a user's device (e.g., provided by a client) through a client library.

[0106] Fig. 6A An example embodiment of a write operation of a storage scheme with local data recovery and server-side encryption according to an example embodiment of the present disclosure is shown. For example, Fig. 6A The illustrated embodiment can be used to implement Figure 4 and Figure 5 Any storage options shown. Fig. 6A The left side shows the data flow between storage system components. Figure 3A The right side of shows the manipulation of data during a write operation.

[0107] Fig. 6A The system shown on the left may include a client 602, one or more servers 604 (collectively referred to as servers), and one or more storage devices 608 (collectively referred to as storage devices). Fig. 6A The operations shown on the right side of are shown in a first group 610A performed by the client 602 and a second group 612A performed by the server 604 .

[0108] The write operation may begin when the storage device 608 and / or the server 604 provides the client 602 with one or more indications 632 indicating data chunk size, compression algorithm, etc. The client 602 may divide the original data 614 into one or more chunks 633 based on the one or more indications 632, for example.

[0109] Back to reference Figure 3A and Figure 3B , data chunks 320 may be substantially the same size, which may be desirable, for example, when storage device 308 is implemented using one or more block-based storage devices. Fig. 6A In the illustrated embodiment, the data chunks 633 may be of different sizes, for example, to take advantage of one or more of the storage devices 608 that may be implemented with a KV interface. Additionally or alternatively, the server 604 may implement a software emulation of a key-value interface (e.g., RocksDB, LevelDB, etc.) on top of one or more block-based storage devices 608. Figure 3A and Figure 3BThe data blocks shown in are shown as having different sizes, which can be considered a subset of variable-sized blocks, but the principles can also be applied to systems where some or all of the storage devices have block-based interfaces.

[0110] In some embodiments, a suggested and / or mandatory chunk size may be determined, for example, based on a chunk size that may be an optimal known size for a particular storage device. For example, for some solid-state drives (SSDs), a chunk size of 128KB may fully utilize the SSD bandwidth. Additionally or alternatively, the storage server may provide the optimal chunk size to the client 602 via a library, and when a user stores an object or other raw data, the client 602 may internally split the object or other raw data into smaller chunks. Additionally or alternatively, the client 602 may analyze the content and dynamically determine the chunk size.

[0111] After chunking the original data 614, the client may individually compress one or more of the data chunks 633 to generate one or more compressed chunks 634. The client 602 may send the compressed chunks 634 to the server 604, and the server 604 may encrypt the one or more compressed chunks 634 to generate one or more compressed and encrypted data chunks 635. The server 604 may perform erasure coding on the one or more compressed and encrypted data chunks 635 to generate one or more parity chunks 636, and store the one or more data chunks 635 and the one or more parity chunks 636 on one or more storage devices 608.

[0112] Figure 6B An example embodiment of a write operation of a storage scheme with local data recovery and client-side encryption according to an example embodiment of the present disclosure is shown. For example, Figure 6B The illustrated embodiment can be used to implement Figure 4 and Figure 5 Any storage options shown. Figure 6B The left side shows the data flow between storage system components. Figure 6B The right side of shows the manipulation of data during a write operation.

[0113] Figure 6B The data flow between the components and / or data operations shown in can be similar to Fig. 6A The embodiments with server-side encryption shown in and elements with reference numbers ending with the same number may be similar. Figure 6BIn the illustrated embodiment, the client 602 may encrypt one or more compressed data chunks 634 to generate one or more compressed and encrypted data chunks 635, as shown in the first group of operations 610B. The client 602 may send the one or more compressed and encrypted data chunks 635 to the server 604, which may perform erasure coding on the one or more compressed and encrypted data chunks 635 to generate one or more parity chunks 636, and store the one or more data chunks 635 and the one or more parity chunks 636 on the one or more storage devices 608, as shown in the group of operations 612B.

[0114] After one or more data chunks 633 have been individually modified (e.g., compressed, encrypted, etc.) and stored as modified data chunks 636 across one or more storage devices 608, each storage device is able to recover the one or more data chunks (e.g., by decrypting and / or decompressing the one or more data chunks) and perform operations on the recovered data chunks. For example, a user, client 602, server 604, etc. may send a request to one or more storage devices 608 to recover the one or more chunks and perform one or more operations (e.g., data selection operations) on the recovered data chunks.

[0115] Fig. 7A An example embodiment of a write operation of a storage scheme with local data recovery according to an example embodiment of the present disclosure is shown. Figure 7B An example embodiment of a read operation with data selection of a storage scheme with local data recovery according to an example embodiment of the present disclosure is shown. For example, Fig. 7A and Figure 7B The illustrated embodiment can be used to implement Figure 4 and Figure 5 Any storage options shown.

[0116] refer to Fig. 7A , the left side shows the flow of data between storage system components, and the right side shows the operations on the data during a write operation. Figure 7B , the left side shows the data flow between storage system components, and the right side shows the operations on the data during a read operation.

[0117] Fig. 7A The write operation shown in can be implemented similar to Fig. 6A Server-side encryption as shown in , or similar to Figure 6B , and elements with reference numerals ending with the same number may be similar. Thus, as part of client operation 710A or part of server operation 712A, data chunks 734 that have been individually compressed may be encrypted to generate compressed and encrypted data chunks 735.

[0118] refer to Figure 7B , one or more computing storage devices 708 may receive one or more requests to perform a data selection operation to read one or more data subsets from one or more data partitions 735 stored on the storage device. The one or more requests may include, for example, one or more expressions to specify the requested data subsets. For example, the request may be received from a client 702 via a server 704.

[0119] To process the one or more requests, the one or more storage devices 708 may perform a set of operations 737 locally on the one or more storage devices 708. Each of the three different storage devices may perform a set of data recovery and data selection operations 737-1, 737-2, and 737-3, respectively, on the corresponding data chunks stored at each device. However, in some embodiments, a single storage device may perform data recovery and data selection or other operations on any number of data chunks stored on the device.

[0120] Each storage device 708 can read a respective data chunk 735 that has been individually compressed and encrypted from the storage medium. Each storage device can decrypt the respective data chunk to generate compressed and decrypted data chunk 734. Each storage device can decompress the respective data chunk to generate recovered data chunk 738. In some embodiments, each recovered data chunk 738 can be identical to a corresponding share of the original data 714. However, in some embodiments, the recovered data chunk 738 can be recovered only to a form that enables the storage device 708 to perform meaningful operations on the recovered data (e.g., some embodiments may be able to perform one or more operations on data chunks that have not been fully decompressed).

[0121] After the data chunks are recovered, each storage device 708 may perform a data selection operation (e.g., scan, filter, etc.) based on, for example, an expression provided with the request to obtain one or more corresponding results 739. One or more storage devices 708 may send the results 739 to the client as one or more requested subsets 740 of the original data 714. Because the client's decompression and / or decryption operations may be bypassed, they are grayed out.

[0122] In some embodiments, if parity chunks 736 are stored on a storage device, one or more of the storage devices 708 may be able to recover the one or more lost data chunks 735. Alternatively or additionally, the server 704 may recover the one or more lost data chunks 735 using one or more parity chunks 736 stored on one or more other storage devices.

[0123] Depending on implementation details, performing data recovery and / or data selection operations at a storage device can reduce the time, bandwidth, power, latency, etc. associated with reading a subset of original data (e.g., a subset of an object) stored in one or more blocks across one or more storage devices.

[0124] Figure 8 An example embodiment of a system architecture of an object storage solution with local data recovery according to an example embodiment of the present disclosure is shown. For example, Figure 8 The system shown can be used to implement the reference Figure 4 , Figure 5 , Fig. 6A , Figure 6B , Fig. 7A , Figure 7B , Fig. 9A and / or Fig. 9B Any scenario described.

[0125] Figure 8 The system shown may include a client 802 and an object storage server cluster 804 connected via a network connection 842. The system may also include one or more storage nodes 806 connected to the object storage server cluster 804 via a storage network 844.

[0126] The client 802 may include data chunking logic 826 and / or compression logic 846, which may be configured to perform data chunking on original data (e.g., one or more objects) before compressing the individual data chunks so that one or more computing storage devices 808 can recover the data chunks to perform operations on the recovered data chunks.

[0127] The object storage server cluster 804 may include encryption logic 847, erasure coding logic 848, data selection logic 849, cluster management logic 850, and / or node and storage device management logic 851. The encryption logic 847 may be used to individually encrypt data chunks (e.g., compressed data) received from the client 802. The erasure coding logic 848 may perform erasure coding on the data chunks on the storage nodes 806 and / or storage devices 808. The data selection logic 849 may perform various operations related to data recovery, data selection, and / or other processing operations performed by the various storage devices 808. For example, the data selection logic 849 may receive a request from the client 802 to read one or more data subsets in a chunk that may be stored on one or more storage devices 808. The data selection logic 849 may forward the request to the corresponding storage nodes 806 and / or storage devices 808, receive and / or aggregate the results from the corresponding storage nodes 806 and / or storage devices 808, and send the aggregated results to the client 802. The cluster management logic 850 may perform housekeeping and / or management functions associated with maintaining the storage server cluster 804. The node and storage device management logic 851 may perform housekeeping and / or management functions associated with maintaining one or more storage nodes 806 and / or storage devices 808.

[0128] Each of the storage nodes 806 may include a processing unit (e.g., a data processing unit (DPU), a CPU, etc.) 852 and one or more computational storage devices 808. The DPU 852 may perform various functions, such as receiving and dispatching requests from clients 802 to read one or more data subsets in chunks that may be stored on one or more storage devices 808. In some embodiments, the DPU 852 may perform data compression, data encryption, erasure coding, etc. on the data chunks received from the object storage server cluster 804 and stored on the one or more computational storage devices 808. In some embodiments, the DPU 852 may aggregate the results of one or more data selection operations performed by the one or more computational storage devices 808 and forward the aggregated results to the object storage server cluster 804 and / or the client 802.

[0129] Computational storage device 808a illustrates an example of a component that may be included in one or more computational storage devices 808. Computational storage device 808a may include a data selection engine 853 and a storage medium 830. The data selection engine 853 may include decryption logic 854 and decompression logic 855, which may be used to decrypt and / or decompress data blocks that have been individually encrypted and / or compressed, respectively, to restore the data blocks to a form that can be operated. The data selection engine 853 may also include data selection logic 856, which may be used to perform data selection or other operations on the restored data blocks. The data selection engine 853 may also include KV logic 857, which may be used to implement the KV interface of the storage device 808a.

[0130] In some embodiments, Figure 8 The system shown may be implemented with a KV interface for some or all of the storage devices 808. Depending on the implementation details, this may facilitate and / or enable data chunking to be implemented with variable chunk sizes. For illustration purposes, Figure 8 The illustrated embodiments may be described as implementing data selection features locally using recovered data chunks on one or more storage devices 808, however, the principles may be applied to any type of processing that may be performed on recovered data chunks.

[0131] Fig. 9A An example embodiment of a read and / or write operation of a storage scheme with local data recovery according to an example embodiment of the present disclosure is shown. For example, Figure 8 The system shown is implemented Fig. 9A For the purpose of illustration, the first set of operations 958A may be assumed to be Figure 8 The second set of operations 959A may be assumed to be performed by the client 802 and the object storage server cluster 804 as shown. Figure 8 One or more storage nodes 806 and / or storage devices 808 are shown to perform, however, in other embodiments, Fig. 9A The operations shown may be performed by any other components.

[0132] refer to Fig. 9A During a write operation (e.g., a put operation), original data 914 (e.g., one or more objects) may be chunked by a client to generate one or more data chunks 933. One or more chunks 933 may be individually compressed by a client to generate one or more compressed chunks 934, which may be sent to an object storage server and individually encrypted by the object storage server to generate one or more compressed and / or encrypted chunks 935. The object storage server may perform erasure coding on one or more compressed and / or encrypted chunks 935 to generate one or more parity chunks 936.

[0133] The object storage server may send the one or more compressed and encrypted chunks 935 and the one or more parity chunks 936 (e.g., via a put operation 960) to one or more storage nodes for storage on one or more storage devices. Thus, after a write operation, the original data 914 (e.g., an object) may be stored in one or more chunks 935 that may have been individually modified (e.g., compressed and / or encrypted) across one or more storage devices.

[0134] During a read operation (e.g., a get operation), for example, in implementations where the storage device may not recover and / or perform operations on data chunks, one or more individually modified data chunks 935 may be read from one or more storage devices. If one or more of the data chunks 935 are lost or corrupted, one or more parity chunks 936 may be used to recover (e.g., by a storage device and / or storage node) the lost and / or corrupted chunks.

[0135] The one or more compressed and / or encrypted chunks 935 may be sent to the object storage server (e.g., via a get operation 962), which may decrypt the one or more compressed and / or encrypted chunks 935 to generate one or more compressed and decrypted chunks 934. The one or more compressed and decrypted chunks 934 may be sent to the client, which may decompress the one or more data chunks 934 to generate decrypted and decompressed data chunks 933 and assemble them back into the original data 914.

[0136] Fig. 9B An example embodiment of a read operation of a storage scheme with local data recovery and data selection operations according to an example embodiment of the present disclosure is shown. For example, Figure 8 The system shown is implemented Fig. 9B For the purpose of illustration, the first set of operations 958B may be assumed to be Figure 8 The second set of operations 959B may be assumed to be performed by the client 802 and / or the object storage server cluster 804 as shown. Figure 8 One or more storage nodes 806 and / or storage devices 808 are shown to perform, however, in other embodiments, Fig. 9B The operations shown may be performed by any other components.

[0137] To begin a read operation (e.g., get operation 963), one or more computational storage devices may receive one or more requests to perform a data selection operation to read one or more data subsets from one or more data chunks 935 stored at one or more storage devices. The one or more requests may include, for example, one or more expressions to specify the requested data subsets.

[0138] To service one or more requests, one or more individually modified data chunks 935 may be read from one or more storage devices. One or more storage devices may individually decrypt one or more data chunks 935 to generate one or more compressed and decrypted data chunks 934. One or more storage devices may individually decompress one or more compressed and decrypted data chunks 934 to generate one or more restored data chunks 938. In some embodiments, each restored data chunk 938 may be identical to a corresponding share of the original data 914. However, in some embodiments, the restored data chunks 938 may only be restored to a form that enables the storage device to perform meaningful operations on the restored data (e.g., some embodiments may be able to perform one or more operations on data chunks that have not been fully decompressed).

[0139] The storage device may perform a data selection operation (e.g., scanning, filtering, etc.) on one or more recovered data chunks 938 to find one or more data subsets 939 (indicated as results R) specified by one or more requests. If the storage device has recovered more than one data chunk and performed a data selection operation on more than one data chunk, the storage device may aggregate the results of the data selection operation to generate an aggregated result 940, which may be sent to the object storage server and the client that sent the request. Additionally or alternatively, the results R (e.g., data subsets) 939 found by the data selection operations of multiple storage devices may be aggregated by the storage nodes and sent to the object storage server and the client that sent the request.

[0140] Table 1 shows some example data that can be stored in a storage system according to an example embodiment of the present disclosure. For illustrative purposes, the data shown in Table 1 is for a real estate listing, but the principles can be applied to any type of data. Each row of Table 1 can correspond to a record with seven entries: record index, square feet of living space, number of bedrooms, number of bathrooms, zip code, year of construction, and list price. Thus, for example, the first eight records can be identified by indexes 1-8, respectively.

[0141] Table 1

[0142]

[0143]

[0144] Fig.10 An embodiment of the distribution of data from Table 1 on three data blocks at three computing storage devices according to an example embodiment of the present disclosure is shown. Fig.10 In the illustrated embodiment, a semicolon is used as a separator between individual records (which may correspond to the rows shown in Table 1), but in other embodiments, other separation techniques may be used.

[0145] refer to Fig.10 , the first two records (identified by indexes 1 and 2) may fit completely into the first data split 1064A stored on the first storage device 1008A. The third record (identified by index 3 and indicated by an entry with a single underline) may be split (e.g., segmented) between the data splits 1064A and 1064B stored on the first and second storage devices 1008A and 1008B, respectively. The fourth and fifth records (identified by indexes 4 and 5) may fit completely into the second data split 1064B stored on the second storage device 1008B. The sixth record (identified by index 6 and indicated by an entry with a single underline) may be split between the data splits 1064B and 1064C stored on the second and third storage devices 1008B and 1008C. The seventh and eighth records (identified by indexes 7 and 8) may fit completely into the third data split 1064C stored on the third storage device 1008C. The ninth record (identified by index 9 and indicated by the entry with a single underline) may be split between a third data chunk 1064C stored on a third storage device 1008C and another chunk on another storage device.

[0146] For purposes of illustration, computational storage devices 1008A, 1008B, and 1008C are shown as being implemented with data recovery logic and / or processing elements as described above, which may enable the storage device to recover individually modified data chunks 1035, such as by decryption (to generate decrypted data chunks 1034) and / or decompression, to generate recovered data chunks 1038, and to perform operations such as data selection operations on recovered data chunks 1038 to obtain a specified data subset 1039 from one or more records in the data chunks stored on the device. However, the principles are not limited to these implementation details, but may be applied to any type of operation that may be performed on any type of data chunks stored on any type of computational storage device. For purposes of illustration, some embodiments described herein may implement fixed-size data chunks (e.g., as may be used with block-based storage devices), however, the principles may also be applied to embodiments that may implement variable-size data chunks (e.g., as may be used with KV storage devices).

[0147] In some embodiments, a record may correspond to an object. In some embodiments described herein, a record (e.g., a JSON object) may be assumed to be smaller than a block, which, depending on implementation details, may ensure that an object may span no more than two blocks. In some embodiments, a delimiter may be implemented as a simple character, such as a semicolon. For example, for a CSV object, a delimiter may be implemented as a carriage return. Additionally or alternatively, one or more delimiters may be determined by a hierarchical structure. Therefore, detecting delimiters may be more complex than simple comparison. For example, for a JSON object, a pair of curly braces ("{...}") may define a JSON object. In addition, in some embodiments, a JSON object may have a nested JSON array, so the outermost pair of curly braces may define a single record. Therefore, a delimiter may be defined by the outermost right curly brace ("}").

[0148] Reference again Fig.10 , records that fit completely into one of the storage devices (e.g., records 1, 2, 4, 5, 7, and 8) can be processed by the corresponding storage device. For example, if a client issues a read request for a data selection operation that returns a subset of the data stored in Table 1 (e.g., a client sends a read request with an expression to return all records (or a portion thereof) with a year of construction after 1980), then records 1, 2, 4, 5, 7, and 8 can be processed directly by the corresponding storage device. However, records 3, 6, and 9 may not be processed locally on the storage device because they are segmented between data partitions of two different storage devices.

[0149] Fig.11 An example embodiment of a storage system in which a server can reconstruct records split between data chunks at different storage devices according to an example embodiment of the present disclosure is shown. Fig.11 The illustrated embodiment may include an object storage server 1104, two storage nodes 1106A and 1106B coupled to the object storage server 1104 via a storage network, and computational storage devices 1108A, 1108B, 1108C, 1108D, and 1108E that may store data chunks, including chunks similar to Fig.10 The records shown.

[0150] refer to Fig.11, each of the storage devices 1108A through 1108E may respectively (directly, or through the storage node on which it resides) send the results 1165 of the data selection operations it may perform on any complete record in its corresponding data split 1164A through 1164E. However, because records 3, 6, and 9 may not be processed locally on the storage device, the object storage server 1104 may reconstruct the split records in one or more aggregation buffers 1166. In some embodiments, each aggregation buffer 1166 may reconstruct the split records between the i-th device and the (i+1)-th device. For example, storage device 1108A may send a first share (also referred to as a fragment) (e.g., index, living space, bedroom, bathroom, zip code, and year of construction) of record 3 located in data split 1164A to object storage server 1104 to be aggregated in first buffer 1166A with a second share (price) of record 3 located in data split 1164B and sent by storage device 1108B. In some embodiments, object storage server 1104 may include N aggregate buffers, where N may be the number of storage devices coupled to object storage server 1104 .

[0151] Object storage server 1104 may perform a select operation on reconstructed records 3, 6, and 9 in aggregate buffers 1166A, 1166B, and 1166C, respectively, to generate result 1167. Thus, between result 1165 sent by each storage device and result 1167 generated from aggregate buffer 1166, object storage server 1104 may obtain all data subsets specified by the request and return these subsets to the client.

[0152] However, depending on implementation details, each of the record fragments sent from storage device 1108 to object storage server 1104 may consume time, bandwidth and / or power, increase latency, reduce utilization of processing resources, etc., and / or may cause object storage server 1104 to become a potential bottleneck.

[0153] Fig.12 An embodiment of a content-aware method for dividing data into blocks and compressing data according to an exemplary embodiment of the present disclosure is shown. Fig.12 In the method shown, Fig.12The uncompressed input data 1268 shown at the top (which may be stored in a buffer) may include one or more records, each of which may end with a delimiter, which in this example may be a semicolon. However, in other embodiments, other delimiters such as brackets, carriage return line feed (CR-LF) sequences, etc. may be used. In some embodiments, one or more records may be included in an array (e.g., a JSON array), a document (e.g., a JSON document), etc. within the input data 1268. If a record ends at the end of an array, document, etc., a different delimiter may be used to indicate the end of the array, document, etc., but Fig.12 The method shown may also interpret this different delimiter as the end of a record.

[0154] During a first pass (operation 1), the method can scan the input data 1268 and use it to generate data chunks until it reaches a default chunk size N. If the last data element in the chunk 1269 is a delimiter, the method can proceed to a second pass (operation 2). If the last data element in the chunk is not a delimiter, the method can modify the size of the chunk so that the end of the chunk is aligned with the end of the delimiter. For example, the method can start at position N+1 in the buffer that holds the data 1268 and scan backward until the delimiter 1270 is found. The method can reduce the size of the chunk (e.g., shrink the chunk) until the delimiter 1270 is the last data element in the chunk. Therefore, the chunk can be reduced by a size equal to the shrink space S, so that the chunk can include only one or more complete records.

[0155] Alternatively, the method may start 1268 at position N+1 in the buffer holding the data and scan forward until a delimiter 1271 is found. The method may extend the size of the chunk until the delimiter 1271 is the last data element in the chunk 1269, such as Fig.12 Therefore, the block can be extended by a size equal to the extension space E, so the block can only include one or more complete records.

[0156] Once the chunk size is determined, the method can proceed to a second pass (operation 2), where a self-contained chunk (e.g., chunk 1269) that may end with a delimiter and include only complete records can be compressed to a length C using one or more compression algorithms to generate compressed data 1272.

[0157] Because the size of the uncompressed data chunks can be reduced or expanded, the size of the uncompressed data chunks 1269 can be variable, and therefore the size of the compressed data chunks 1272 can be variable.

[0158] Whether the approach is to reduce or extend the length of the chunks may depend on various factors, such as the maximum and / or optimal data size of an object or key-value store to which the chunks may be sent, or the block size of a block-based store to which the chunks may be sent. In the case of an object or key-value store, the resulting compressed, self-contained data chunks 1272 may be stored without further modification and / or processing.

[0159] Depending on the implementation details, Fig.12 The content-aware data partitioning method shown can improve Fig.11 Operation of the storage system shown, for example, because fewer or no records can be split between blocks on different storage devices, fewer or no record fragments can be passed between storage devices, storage nodes, storage servers, etc. to enable components to recover data chunks with complete records and / or perform operations on complete records.

[0160] For example, you can use Figure 4 , Figure 5 and / or any of the systems shown in FIG. 9 to implement Fig.12 The compression method shown, in which the chunking logic 426, 526 and / or 826 may include content-aware logic that is configured to determine the chunk size based on finding boundaries (e.g., delimiters) of records or other data units in the uncompressed input data 1268.

[0161] although Fig.12 The illustrated approach may improve the operation of a storage system, but in some embodiments, depending on implementation details, the overhead associated with using two passes may increase the processing burden, especially when scanning relatively large objects.

[0162] Some additional content-aware data chunking techniques according to example embodiments of the present disclosure may integrate the chunking operation with another operation, such as a data compression operation that may scan the data to be chunked. Depending on implementation details, this may improve the efficiency of the chunking operation because it may take advantage of the scanning operation already performed for compression. Thus, in some embodiments, it may reduce or eliminate the overhead associated with two scan passes.

[0163] Fig.13 An embodiment of a method for compressing data according to an exemplary embodiment of the present disclosure is shown. Fig.14 Show that it can be Fig.13 The illustrated method is an embodiment of the operations performed. Fig.13 and Fig.14 The method shown may be based on, for example, a compression algorithm such as LZ77. Fig.13As shown at the top of , the method can operate on an input data stream that may pass through a lookahead buffer 1374 and a window 1373 (which may also be implemented as a buffer). The portion (also referred to as a part) of the input data stream before the lookahead buffer 1374 may be indicated as an invisible portion 1375 because it may not be sent to the logic that performs compression. Depending on how much data has passed through the lookahead buffer 1374 and the window 1373, there may be another portion of invisible data after the window 1373.

[0164] Fig.13 The method shown can be performed as follows Fig.14 The following operations are shown. Fig.13 and Fig.14 The illustrated embodiments may be implemented with data elements of any size, configuration, etc., such as bytes, words, nibbles, etc.

[0165] (1) In the first operation 1477-1, the method may set the encoding position to the first data element of the input data stream, such as Fig.13 as shown at the top of the .

[0166] (2) In a second operation 1477-2, the method may scan the input data stream to find a longest match of length L between one or more data elements (which may be referred to as literals) in the look-ahead buffer 1374 and one or more data elements in the window 1373, such as Fig.13 Shown at top.

[0167] (3) At the third operation 1477-3, if a match is found at the second operation (2) 1477-2, the method may output a pointer P, which may include, for example, an offset from the encoding position indicating the start position of the matching data in the window 1373, and a length L indicating the number of matched data elements, such as Fig.13 This method can move the encoding position and window 1373 forward by L data elements, as shown in FIG. Fig.13 The method may proceed to the fifth operation (5) 1477-5. Depending on the length L, this may result in all or part of the invisible portion 1375 of the input data stream entering the look-ahead buffer 1374, and the portion 1375 of the data length L previously in the window 1373 becoming invisible (e.g., inaccessible to the logic executing the method).

[0168] (4) In the fourth operation 1477-4, if no match is found in the second operation (2) 1477-2, the method may output a null pointer (e.g., a pointer with an offset and / or length L of zero) and the next data element (literal) in the lookahead buffer 1374. The method may move the encoding position and window 1373 forward by one data element.

[0169] (5) In the fifth operation 1477-5, if the look-ahead buffer 1374 is not empty, the method may return to the second operation (2) 1477-2. Otherwise, the method may terminate.

[0170] therefore, Fig.13 and Fig.14 The illustrated compression method may operate by checking whether a sequence of one or more literals in the look-ahead buffer 1374 has ever appeared within the range of the window 1373 and replacing it with a pointer (e.g., index) indicating the instance of the sequence in the window 1373. The output data stream generated by the compression operation may be a series of literal data elements (e.g., bytes) and pointers that may indicate sequences of data elements that are already in the output.

[0171] In some embodiments, a record may include any complete data structure that can be used and / or manipulated as a unit (e.g., a row in a database). In some embodiments, a delimiter may be implemented with anything that can indicate a record boundary, such as one or more data elements (semicolons, parentheses, carriage return line feed (CR-LF) sequences, etc.). In some embodiments, a delimiter may be implemented with a data structure such as an index table that may indicate where a record ends.

[0172] because Fig.13 and Fig.14 The compression method shown can scan the input data as part of its normal operation, so the block operation according to the example embodiment of the present disclosure can be integrated into Fig.13 and Fig.14 In the compression method shown, a data scan is performed to utilize the compression operation.

[0173] Fig.15 An embodiment of an integrated blocking and compression scheme according to an example embodiment of the present disclosure is shown.

[0174] like Fig.15As shown at the top, the input data stream may pass through a lookahead buffer 1574 and a window 1573 (which may also be implemented as a buffer). The portion of the input data stream before the lookahead buffer 1574 may be indicated as an invisible portion 1575, as it may not be sent to the logic that performs compression. Depending on how much data has passed through the lookahead buffer 1574 and the window 1573, there may be another portion of invisible data after the window 1573.

[0175] Fig.16 It shows that the example embodiments according to the present disclosure can be used Fig.15 The illustrated scheme implements an embodiment of an integrated segmentation and compression method.

[0176] Fig.15 and / or Fig.16 The illustrated embodiments may be implemented with data elements of any size, configuration, etc., such as bytes, words, nibbles, etc. Fig.15 and / or Fig.16 The illustrated embodiment may use the current chunk size (c), an early termination indicator (e.g., an early termination bit) (e), and a record delimiter length (l). In some embodiments, Fig.15 The scheme shown can be implemented as Fig.16 The following operations are shown.

[0177] (1) In the first operation 1678-1, the method may set the encoding position to the first data element of the input data stream, such as Fig.15 In some embodiments, the current chunk size (c), the early termination bit (e), and / or the record delimiter length (l) may be initialized to zero. In some embodiments, the initial encoding position may correspond to the beginning of the data chunk, such as Fig.15 shown in the middle and bottom.

[0178] (2) In a second operation 1678-2, the method may scan the input data stream to find a longest match of length L between one or more data elements in the look-ahead buffer 1574 and one or more data elements in the window 1573, such as Fig.15 As shown at the top. While scanning the input data stream for the longest match, the method may also look for a match for the recorded delimiter. If a match for the delimiter is found, the method may, for example, keep the first instance of the delimiter match in window 1573 (e.g., record its position) and set the delimiter length l to the length of the found delimiter match.

[0179] (3) In the third operation 1678-3, if the longest match is not found in the second operation (2) 1678-2, the method can output a null pointer (e.g., a pointer with an offset and / or length L of zero) and the next data element (literal) in the lookahead buffer 1574 to generate a compressed output data stream. The method can move the encoding position and window 1573 forward by one data element. The method can also set the length L to 1, which can indicate that the current block size will increase by one data element output from the lookahead buffer 1574.

[0180] (4) In the fourth operation 1678-4, if the early termination bit (e) is not set and the current block size (c+L) is greater than or equal to the default block size (N), the method may set the early termination bit to 1 (e=1).

[0181] (5) In the fifth operation 1678-5, if the early termination bit is set (e=1) and a delimiter match is found (e.g., l!=0), the method may set L to be equal to l (l is greater than 1) and update the pointer P with the first matching delimiter (e.g., with an offset from the encoding position indicating the starting position of the first matching delimiter in window 1573, and a length l indicating the length of the matching delimiter).

[0182] (6) In the sixth operation 1678-6, if a match is found (e.g., longest match or delimiter match), the method can be, for example, as follows Fig.15 The position and length of the longest match (e.g., a normal match) shown in the middle or Fig.15 The bottom portion shows an output pointer P based on the position and length of the second delimiter match (eg, early termination) to generate a compressed output data stream.

[0183] (7) In the seventh operation 1678-7, the method may move the encoding position and window 1573 forward by L data elements, where L may be the length of the longest match, such as Fig.15 or L can be the length l of the delimiter match, as in Fig.15 After the seventh operation, depending on the lengths of L and l, the data stream may have one or more invisible portions 1575A, 1575B, and / or 1575C.

[0184] (8) In the eighth operation 1678-8, if the look-ahead buffer 1574 is not empty, the method may return to the second operation (2) 1678-2. Otherwise, the end of the block may be determined by the end of the delimiter, such as Fig.15 The coding position at the bottom is shown.

[0185] In some embodiments, similar to Fig.13 and Fig.14 The method shown, Fig.15 The scheme shown and / or Fig.16 The method shown can initially perform compression by finding the longest match until a default block size N is reached. However, when approaching the default block size, Fig.15 and / or Fig.16 The illustrated embodiment may compress the input data record by record until it exceeds the default chunk size. For example, in some embodiments, the method may find the smallest chunk that is larger than the default chunk size (and may terminate with a delimiter). In some embodiments, this may be referred to as the smallest maximum chunk. As the method scans the input data stream, Fig.15 The start of the blocks shown in the middle and bottom may be gradually moved away from the encoding position.

[0186] Depending on the implementation details, Fig.15 and / or Fig.16 The content-aware data chunking technique shown in can improve Fig.11 Operation of the storage system shown in , for example, because fewer or no records can be split between chunks on different storage devices, and therefore, fewer or no record fragments can be passed between storage devices, storage nodes, storage servers, etc. to enable components to recover data chunks with complete records and / or perform operations on complete records.

[0187] Depending on the implementation details, Fig.15 and / or Fig.16 The illustrated embodiments may also improve the efficiency of data blocking operations because it may take advantage of scan operations already performed for compression. Thus, in some embodiments, they may reduce or eliminate the overhead associated with two scan passes.

[0188] Fig.17 Another embodiment of an integrated segmentation and compression scheme according to an example embodiment of the present disclosure is shown.

[0189] Fig.18 It shows that the example embodiments according to the present disclosure can be used Fig.17 The illustrated scheme implements an embodiment of an integrated segmentation and compression method.

[0190] Fig.17 The scheme shown may include a scheme similar to that for Fig.15 The components and / or implementations described in the illustrated embodiments are similar to those described for Fig.15 The operations described in the illustrated embodiments are similar in that elements having reference numerals ending with the same number may be similar. Fig.17In the illustrated embodiment, the method may determine the chunk size when encountering the first record, rather than compressing the input data record by record. For example, in some embodiments, the method may find the largest chunk that is smaller than the default chunk size (and may terminate with a delimiter). In some embodiments, this may be referred to as the largest minimum chunk.

[0191] like Fig.17 As shown at the top, the input data stream may pass through a lookahead buffer 1774 and a window 1773 (which may also be implemented as a buffer). The portion of the input data stream before the lookahead buffer 1774 may be represented as an invisible portion 1775, as it may not be sent to the logic that performs compression. Depending on how much data has passed through the lookahead buffer 1774 and the window 1773, there may be another portion of invisible data after the window 1773.

[0192] Fig.17 The illustrated embodiment may be implemented as follows Fig.18 One or more of the following operations shown. Fig.17 and / or Fig.18 The illustrated embodiments may be implemented with data elements of any size, configuration, etc., such as bytes, words, nibbles, etc. Fig.17 and / or Fig.18 The illustrated embodiment may use the current chunk size (c), an early termination indicator (eg, an early termination bit) (e), and a record delimiter length (l).

[0193] (1) In the first operation 1879-1, the method may set the encoding position to the first data element of the input data stream, such as Fig.17 In some embodiments, the current chunk size (c), the early termination bit (e), and / or the record delimiter length (l) may be initialized to zero.

[0194] (2) In a second operation 1879-2, the method may scan the input data stream to find a longest match of length L between one or more data elements in the look-ahead buffer 1774 and one or more data elements in the window 1773, such as Fig.17 As shown at the top. While scanning the input data stream for the longest match, the method may also look for a match for the delimiter of the record. If a match for the delimiter is found, the method may keep the first instance of the delimiter match, such as in window 1773, and set the delimiter length l to the length of the delimiter match found.

[0195] (3) In the third operation 1879-3, if the longest match is not found in the second operation (2) 1879-2, the method can output a null pointer (e.g., a pointer with an offset and / or length L of zero) and the next data element (literal) in the lookahead buffer 1774 to generate a compressed output data stream. The method can move the encoding position and window 1773 forward by one data element. The method can also set the length L to 1, which can indicate that the current block size will increase by one data element output from the lookahead buffer 1774.

[0196] (4) In the fourth operation 1879-4, if the current chunk size (c+L) is greater than or equal to the default chunk size (N), the method may set the early termination bit to 1 (e=1).

[0197] (5) In the fifth operation 1879-5, if the early termination bit is set (e=1) and a delimiter match is found (e.g., l!=0), the method may set L to be equal to l (l is greater than 1) and update the pointer P with the first matching delimiter (e.g., with an offset indicating the starting position of the first matching delimiter in window 1773 from the encoding position, and a length l indicating the length of the matching delimiter).

[0198] (6) In the sixth operation 1879-6, if a match is found (e.g., longest match or delimiter match), the method can be based on, for example, Fig.17 The position and length of the longest match (e.g., a normal match) shown in the middle or Fig.17 The delimiter shown at the bottom matches (eg, terminates early) the position and length of the output pointer P to generate a compressed output data stream.

[0199] (7) In the seventh operation 1879-7, the method may move the encoding position and window 1773 forward by L data elements, where L may be as follows: Fig.17 The normal operation in the middle shows the length of the longest match, or L can be as follows Fig.17 The delimiter shown in the early termination operation at the bottom matches a length of l. After the seventh operation, depending on the lengths of L and l, the data stream may have one or more invisible portions 1775A, 1775B, and / or 1775C.

[0200] (8) In the eighth operation 1878-8, if the early termination bit is not set (e.g., e!=1) and the look-ahead buffer 1774 is not empty, the method may return to the second operation (2) 1879-2. Otherwise, the end of the block may be determined by the end of the delimiter, such as Fig.17 The coding position at the bottom is shown.

[0201] In some embodiments, because Fig.17 The scheme shown and / or Fig.18 The method shown can determine the chunk size (maximum and minimum chunks) when encountering the first record, so the chunk size can be matched based on the first delimiter, such as Fig.17 As this method scans the input data stream, such as Fig.17 The start of the blocks shown in the middle and bottom may be gradually moved away from the encoding position.

[0202] Depending on the implementation details, Fig.17 and / or Fig.18 The illustrated embodiment can improve the efficiency of data blocking operations because it can take advantage of scan operations already performed for compression. Thus, in some embodiments, it can reduce or eliminate the overhead associated with two scan passes.

[0203] Fig.19 Another embodiment of the integrated segmentation and compression method according to an exemplary embodiment of the present disclosure is shown. For example, Fig.19 The illustrated embodiment can be used with Fig.15 and / or Fig.17 One or both of the embodiments shown may be used together. In some embodiments, Fig.19 The method shown can implement minimum maximum blocking and / or maximum minimum blocking. For example, depending on the implementation details, Fig.19 The illustrated embodiment may determine the smallest maximum chunk when the longest match does not contain a delimiter, and determine the largest minimum chunk when the longest match contains a delimiter.

[0204] refer to Fig.19 , the method may begin at operation 1979-1. At operation 1979-2, which may be the beginning of a repeating sequence, the method may initialize the chunk size (c) and / or the early termination bit (e) to, for example, zero. At operation 1979-3, the method may initialize a pointer P and a length L. At operation 1979-4, the method may search for the longest match (c, L) in the look-ahead buffer.

[0205] In operation 1979-5, if the longest match is found, the method can proceed along Fig.19Take the first main branch on the left side, as shown below. In operation 1979-6, the method can set the pointer P to the start of the longest match and set the length L to the length of the longest match. In operation 1979-7, the method can determine whether the length of the current block size is greater than or equal to the default block size N (for example, c+L≥N). If the current block size is not greater than or equal to the default block size N, the method can proceed to operation 1979-11 as described below. However, if the current block size is greater than or equal to the default block size, the method can initialize the record delimiter length (l) in operation 1979-8. In operation 1979-9, if the method does not find the first delimiter match (c, l) in the look-ahead buffer, the method can proceed to operation 1979-11 as described below. However, if at operation 1979-9 the method finds the first delimiter match (c, l) in the look-ahead buffer, the method may proceed to operation 1979-10, where the method may set the early termination bit to 1 (e=1), set the pointer P to the beginning of the first delimiter match in the sliding window, and / or set the length L to the length of the first delimiter match. At operation 1979-11, the method may output the pointer P and the length l. The method may proceed to operation 1979-18 starting with Fig.19 Public branches on the right.

[0206] In operation 1979-5, if the longest match is not found, the method can be as follows Fig.19 The method may then proceed to the second main branch from the center of the block. At operation 1979-12, the method may set pointer P to the value of the null pointer and set length L to 1. At operation 1979-13, the method may output null pointer P and the first byte of the lookahead buffer. At operation 1979-14, the method may determine whether the length of the current block size is greater than or equal to the default block size N (e.g., c+L≥N). If the current block size is not greater than or equal to the default block size N, the method may proceed to the step starting from operation 1979-18. Fig.19 However, if at operation 1979-14, the current chunk size is greater than or equal to the default chunk size, the method may initialize the record delimiter length (l) at operation 1979-15. At operation 1979-16, if the method does not find the first delimiter match (c, l) in the look-ahead buffer, the method may proceed to step 1979-18. Fig.19 However, if at operation 1979-16 the method finds the first delimiter match (c, l) in the lookahead buffer, the method may proceed to operation 1979-17, where the method may set the early termination bit to 1 (e=1). The method may proceed to operation 1979-18 starting with Fig.19 Public branches on the right.

[0207] Fig.19 The common branch on the right side can be performed as follows. In operation 1979-18, the block size c can be updated by adding a length L (e.g., c=c+L), where L can be, for example, the length of the longest match, the length of the first delimiter match, or 1. In operation 1979-19, the encoding position and window can be moved by a distance L (e.g., L bytes). In operation 1979-20, if the early termination bit (e=0) is not set, the method can return to operation 1979-3 to start another iteration of the repeating sequence. However, if it is determined that the early termination bit (e=1) is set, then in operation 1979-21, the method can indicate that the end of the current block has been found. In operation 1979-22, the method can determine whether the look-ahead buffer is empty. If the look-ahead buffer is not empty, the method can proceed to operation 1979-23, where the early termination bit e can be reset to zero, and the method can return to operation 1979-3 to start another iteration of the repeating sequence. However, if the look-ahead buffer is empty at operation 1979-22, the method may terminate at operation 1979-24.

[0208] although Fig.13 , Fig.14 , Fig.15 , Fig.16 , Fig.17 , Fig.18 and / or Fig.19 The illustrated embodiments may be described in the context of an input data stream and / or a stream-based compression algorithm, but the principles may also be applied to any other type of input and / or compression algorithm, such as batch-based (e.g., file-based) input data and / or block-, file-based, etc. compression algorithms.

[0209] Table 2 shows an example embodiment of a pseudo code of an integrated segmentation and compression method according to an example embodiment of the present disclosure. For example, the embodiment shown in Table 2 can be used with Fig.15 and / or Fig.17 One or two of the embodiments shown in Table 2 can be used together. In some embodiments, the method shown in Table 2 can implement minimum maximum block and / or maximum minimum block. For example, depending on the implementation details, the embodiment shown in Table 2 can determine the minimum maximum block when the longest match does not contain a delimiter, and determine the maximum minimum block when the longest match contains a delimiter.

[0210] Table 2

[0211]

[0212] Fig. 20An embodiment of a system with integrated chunking and compression according to an example embodiment of the present disclosure is shown. For example, Fig. 20 The system shown can be used to implement the reference Fig.13 , Fig.14 , Fig.15 , Fig.16 , Fig.17 , Fig.18 and / or Fig.19 Any of the techniques described.

[0213] refer to Fig. 20 , the system may include a host 2024 and a computing storage device 2008 that may communicate via a connection 2022. The host 2024 may include content-aware data chunking and compression logic 2090 (which may be referred to as chunking and compression logic, integrated chunking and compression logic, or integrated logic). The chunking and compression logic may be configured to provide one or more data chunks to the storage device 2008. For example, the data chunking and compression logic 2090 may divide an object or other raw data into one or more data chunks before compression so that the storage device 2008 can decompress the data chunks and perform operations on the decompressed data chunks.

[0214] Because the chunking and compression logic 2090 is content aware, in some embodiments it can divide the data into chunks based on the boundaries between records. For example, the chunking and compression logic 2090 can look for record delimiters in the data to divide the data into chunks of variable size so that a chunk can end with a record delimiter and therefore end with a complete record.

[0215] In some embodiments, the block and compression logic 2090 may include one or more look-ahead buffers, window buffers, etc., and may implement Fig.13 , Fig.14 , Fig.15 , Fig.16 , Fig.17 , Fig.18 and / or Fig.19 Any of the technologies described

[0216] Computational storage device 2008 may include decompression logic 2028, one or more processing elements 2029, and storage medium 2030. Decompression logic 2028 may be configured to decompress compressed data blocks into a form on which one or more processing elements 2029 may perform operations. One or more processing elements 2029 may be configured to perform any type of operation, such as data selection (e.g., scanning, filtering, etc.), computational acceleration, chart processing, graphics processing, machine learning, etc. Storage medium 2030 may be used to store any data, including one or more modified data blocks sent by host 2024.

[0217] In some embodiments, the decompression logic 2028 and / or one or more processing elements 2029 may be configured to read and decompress one or more data blocks from the storage medium 2030 in response to a request received at the storage device 2008 that may include a query (e.g., an expression) and return a specified subset of data, or perform any other operation on the recovered data blocks.

[0218] In some embodiments, the computing storage device 2008 may also include indication logic that may be configured to provide one or more indications to the content-aware data chunking and compression logic 2090. The one or more indications may include information that may be used to determine how to divide the original data into chunks, as described above with reference to Figure 5 The embodiment shown is described.

[0219] Host 2024 can be used as above Figure 4 The storage device 2008 may be implemented in any form factor, using any connector configuration and any storage medium, as described above with reference to FIG. Figure 4 The host 2024 and the storage device 2008 may communicate using any type of storage interface and / or protocol and may be implemented in whole or in part with a server chassis, a server rack, a data room, a data center, an edge data center, a mobile edge data center, and / or any combination thereof, and / or used in conjunction with them, as described above with reference to Figure 4 The communication connection 2022 may be implemented using any interconnect and / or network interface and / or protocol, as described above with reference to Figure 4 Any of the functions described in the embodiment shown, including any logic, such as content-aware data segmentation and compression logic 2090, decompression logic 2028, one or more processing elements 2029, etc., can be implemented in hardware, software, or a combination thereof, as described above with reference to Figure 4 In some embodiments, one or more of the decompression logic 2028, the processing element 2029, etc. may be integrated with one or more other components of the storage device, such as a storage device controller, a flash translation layer (FTL), etc. The content-aware data chunking and compression logic 2090 and the decompression logic 2028 may implement any compression and / or decompression technology, such as LZ77, gzip, Snappy, etc.

[0220] Fig.21 An embodiment of a content-aware scheme for dividing data into blocks and encrypting the data according to an exemplary embodiment of the present disclosure is shown. Fig.21 In the embodiment shown, Fig.21The unencrypted input data 2168 shown at the top (which may be stored in a buffer) may include one or more records, each of which may end with a delimiter, which in this example may be a semicolon. However, in other embodiments, other delimiters such as brackets, a carriage return line feed (CR-LF) sequence, etc. may be used. In some embodiments, one or more records may be included in an array (e.g., a JSON array), a document (e.g., a JSON document), etc. within the input data 2168. If a record ends at the end of an array, document, etc., a different delimiter may be used to indicate the end of the array, document, etc., but Fig.21 The illustrated embodiment may also interpret this different delimiter as the end of a record.

[0221] During the first pass (operation 1), the method can scan the input data 2168 and use it to generate data chunks until it reaches a default chunk size N. If the last data element in the chunk 2169 is a delimiter, the method can proceed to the second pass (operation 2). If the last data element in the chunk is not a delimiter, the method can modify the size of the chunk so that the end of the chunk is aligned with the end of the delimiter. For example, the method can start at position N+1 in the buffer that holds the data 2168 and scan backward until the delimiter 2170 is found. The method can reduce the size of the chunk (e.g., shrink the chunk) until the delimiter 2170 is the last data element in the chunk. Thus, the chunk can be reduced by the size of the shrink space S so that the chunk can include only one or more complete records.

[0222] Alternatively, the method may start at position N+1 in the buffer holding data 2168 and scan forward until delimiter 2171 is found. The method may extend the size of the chunk until delimiter 2171 is the last data element in chunk 2169, such as Fig.21 Therefore, the block can be extended by a size equal to the extension space E, so the block can only include one or more complete records.

[0223] In some embodiments, once the chunk size is determined, the method may proceed to a second pass (operation 2), where a self-contained chunk (e.g., chunk 2169) that may end with a delimiter and include only complete records may be encrypted using one or more encryption algorithms to generate encrypted data 2172.

[0224] In some embodiments, because the size of the unencrypted data blocks can be reduced or expanded, the size of the unencrypted data blocks 2169 can be variable, and therefore, the size of the encrypted data blocks 2172 can be variable.

[0225] Whether the approach is to reduce or extend the length of the chunks may depend on various factors, such as the maximum and / or optimal data size of an object or key-value store to which the chunks may be sent, or the block size of a block-based store to which the chunks may be sent. In the case of an object or key-value store, the resulting encrypted, self-contained data chunks 2172 may be stored without further modification and / or processing.

[0226] Depending on the implementation details, Fig.21 The content-aware data segmentation embodiment shown can improve Fig.11 Operation of the storage system shown, for example, because fewer or no records can be split between chunks on different storage devices, fewer or no record fragments can be transferred between storage devices, storage nodes, storage servers, etc. to enable components to recover data chunks with complete records and / or perform operations on complete records.

[0227] For example, you can use Figure 4 , Figure 5 and / or any system shown in FIG. 9 to implement Fig.21 The encryption scheme shown, in which the chunking logic 426, 526 and / or 826 may include content-aware logic that is configured to determine the chunk size based on finding boundaries of records or other data units (e.g., delimiters) in the unencrypted input data 2168.

[0228] although Fig.21 The illustrated embodiments may improve the operation of a storage system, but in some embodiments, depending on implementation details, the overhead associated with using two passes may increase the processing burden, especially when scanning relatively large objects.

[0229] Some additional content-aware data chunking techniques according to example embodiments of the present disclosure may integrate the chunking operation with an encryption operation that may scan the data to be chunked. Depending on implementation details, this may improve the efficiency of the chunking operation because it may leverage the scanning operation already performed for encryption. Thus, in some embodiments, it may reduce or eliminate the overhead associated with two scan passes.

[0230] Fig. 22 An example embodiment of a block cipher scheme that can be used for encryption data technology according to an example embodiment of the present disclosure is shown. For example, Fig. 22 The illustrated embodiment may be based on the AES algorithm. The data to be encrypted may be received as blocks (e.g., 128-bit blocks) of plaintext input data 2291, processed in a series of chained operations including pre-round transformation 2292, using a round key K generated by key expansion logic 2296, and then encrypted. 0 ,K1 ,…,K R One or more AES encryption rounds 2293-1, ..., 2293-N r , and output as a corresponding block (e.g., a 128-bit block) of ciphertext output data 2294. For example, the number of rounds R may be determined by the length (e.g., 128, 192, or 256 bits) of a cipher key 2295 used for the encryption round. In some embodiments, the size of the ciphertext output data block 2294 may be the same as the size of the plaintext input data block 2291.

[0231] In some embodiments, the terms plaintext and / or ciphertext refer not only to actual text, but also to any data in any form, such as images, audio, video, etc.

[0232] Because you can use Fig. 22 The block cipher scheme shown is used to scan the input data for encryption, so according to an exemplary embodiment of the present disclosure, Fig. 22 The illustrated embodiment may be integrated with content-aware data chunking techniques. Depending on implementation details, this may improve the efficiency of the chunking operation because it may take advantage of scanning operations that may already be performed for encryption. Thus, in some embodiments, the overhead associated with two scan passes may be reduced or eliminated because the encryption scan and the delimiter scan may be performed simultaneously.

[0233] Fig.23 An embodiment of an integrated data partitioning and encryption scheme according to an example embodiment of the present disclosure is shown. Fig.24 An embodiment of an integrated data segmentation and encryption method according to an exemplary embodiment of the present disclosure is shown. For example, Fig.24 The illustrated embodiment can be used with Fig.23 Use with the scheme shown.

[0234] exist Fig.23 In the scheme shown, Fig.23 The unencrypted input data stream 2368 shown at the top may include one or more records, each of which may end with a delimiter, which in this example may be a semicolon. However, in other embodiments, other delimiters such as brackets, carriage return line feed (CR-LF) sequences, etc. may be used. In some embodiments, one or more records may be included in an array (e.g., a JSON array), a document (e.g., a JSON document), etc. within the input data 2368. If a record ends at the end of an array, document, etc., a different delimiter may be used to indicate the end of the array, document, etc., but Fig.23 The method shown may also interpret this different delimiter as the end of a record.

[0235] Fig.23 The scheme shown can be used Fig.24 One or more of the following operations are shown to implement.

[0236] (1) In the first operation 2497-1, the method can scan the input data stream 2368 and convert such Fig. 22 The block cipher shown is applied to one or more shares (also referred to as portions) 2398 of the input data stream 2368, such as Fig.23 As shown. The block cipher can be applied to the input data stream 2368 on a block-by-block basis, where the size of one or more portions 2398 can correspond to the block size used by the block cipher, which in turn can correspond to the key length used by the block cipher. The input data stream 2368 can be encrypted using, for example, a block cipher until a default block size N is reached. In some embodiments, the default block size N can be set to an even multiple of the block size (e.g., key length) used by the block cipher.

[0237] (2) In the second operation 2497-2, if the last data element of the current data block is not a delimiter, the method may continue to scan the input data stream 2368 to find the delimiter 2399 and extend the data block by an amount E until the delimiter 2399 is the last data element of the block, such as Fig.23 as shown at the top of the .

[0238] (3) In the third operation 2497-3, if the end of the data block is not aligned with the end of the block used by the block cipher, the method may pad the data block with a padding element "p" to further extend the block 23100 by an amount p so that the end of the block is aligned with the end of the block, such as Fig.23 as shown at the bottom of the .

[0239] (4) In a fourth operation 2497-4, if the block has been extended, the block cipher may be applied to the block including the portion E and p (if any).

[0240] (5) In a fifth operation 2497-5, the resulting encrypted data block 23101 may be stored in, for example, a KV drive, and the encrypted data block 23101 may have a variable length based on the position of the delimiter 2399 in the input data stream.

[0241] In some embodiments, Fig.23 and / or Fig.24The illustrated embodiment may continue, for example, until the end of a user object in an input data stream is reached. An object may be divided and encrypted into one or more encrypted data chunks. The size of the encrypted data chunks may be variable because (a) the size of the plaintext input data may be variable, (b) different default chunk sizes N may be used, and / or (c) chunks may be extended to include delimiters and / or one or more padding elements. In the case of an object or key-value storage device, the resulting encrypted self-contained chunks of data may be stored without further modification and / or processing.

[0242] Depending on the implementation details, Fig.23 and / or Fig.24 The content-aware data segmentation embodiment shown in can improve such Fig.11 , for example, because fewer or no records may be split between chunks on different storage devices, and therefore, fewer or no record fragments may be transferred between storage devices, storage nodes, storage servers, etc. to enable a component to recover data chunks with complete records and / or perform operations on complete records. In addition, because Fig.23 and / or Fig.24 The embodiment shown in can utilize the scan operations already performed for encryption to perform data segmentation, so the overhead associated with two scan passes can be reduced or eliminated.

[0243] For illustration purposes, Fig. 22 , Fig.23 and / or Fig.24 The illustrated embodiments may be described in the context of a block cipher such as AES. However, these principles may also be applied to schemes using a stream cipher such as RC4.

[0244] Fig.25 An embodiment of a system with integrated block segmentation and encryption according to an example embodiment of the present disclosure is shown. For example, Fig.25 The system shown can be used to implement the reference Fig. 22 , Fig.23 and / or Fig.24 Any technology described.

[0245] refer to Fig.25, the system may include a host 2524 and a computing storage device 2508 that may communicate via a connection 2522. The host 2524 may include content-aware data chunking and encryption logic 25101 (which may be referred to as chunking and encryption logic, integrated chunking and encryption logic, or integrated logic). The chunking and encryption logic may be configured to provide one or more data chunks to the storage device 2508. For example, the data chunking and encryption logic 25101 may divide an object or other raw data into one or more data chunks before encryption so that the storage device 2508 can decrypt the data chunks and perform operations on the decrypted data chunks.

[0246] Because the chunking and encryption logic 25101 is content aware, in some embodiments it can divide the data into chunks based on the boundaries between records. For example, the chunking and encryption logic 25101 can look for record delimiters in the data to divide the data into chunks of variable size so that a chunk can end with a record delimiter and therefore end with a complete record.

[0247] Computational storage device 2508 may include decryption logic 2528, one or more processing elements 2529, and storage medium 2530. Decryption logic 2528 may be configured to decrypt encrypted data blocks into a form on which one or more processing elements 2529 may perform operations. One or more processing elements 2529 may be configured to perform any type of operation, such as data selection (e.g., scanning, filtering, etc.), computational acceleration, chart processing, graphics processing, machine learning, etc. Storage medium 2530 may be used to store any data, including one or more modified data blocks sent by host 2524.

[0248] In some embodiments, the decryption logic 2528 and / or one or more processing elements 2529 may be configured to read and decrypt one or more data blocks from the storage medium 2530 in response to a request received at the storage device 2508 that may include a query (e.g., an expression) and return a specified subset of data, or perform any other operation on the recovered data blocks.

[0249] In some embodiments, the computing storage device 2508 may also include indication logic, which may be configured to provide one or more indications to the content-aware data chunking and decryption logic 25101. The one or more indications may include information that can be used to determine how to divide the original data into chunks, as described above with reference to Figure 5 The embodiment shown is described.

[0250] Host 2524 can use the above Figure 4The storage device 2508 may be implemented in any form factor, using any connector configuration, and any storage medium, as described above with reference to Figure 4 The host 2524 and the storage device 2508 may communicate using any type of storage interface and / or protocol and may be implemented in whole or in part with a server chassis, a server rack, a data room, a data center, an edge data center, a mobile edge data center, and / or any combination thereof, and / or used in conjunction with them, as described above with reference to Figure 4 The communication connection 2522 may be implemented using any interconnect and / or network interface and / or protocol, as described above with reference to Figure 4 Any of the functions described in the embodiment shown, including any logic, such as content-aware data segmentation and encryption logic 25101, decryption logic 2528, one or more processing elements 2529, etc., can be implemented in hardware, software, or a combination thereof, as described above with respect to Figure 4 In some embodiments, one or more of the decryption logic 2528, the processing element 2529, etc. may be integrated with one or more other components of the storage device, such as a storage device controller, a flash translation layer (FTL), etc. The content-aware data chunking and encryption logic 25101 and the decryption logic 2528 may implement any encryption and / or decryption technology, such as the Data Encryption Standard (DES), AES, etc.

[0251] Although some embodiments may be described in the context of integrated data segmentation and compression, encryption, etc. for use with computational storage devices, these principles are not limited to any particular application and may be applied to any data segmentation application and for any type of device including accelerators, network interface cards (NICs), etc.

[0252] For illustrative purposes, embodiments that can implement integrated chunking and compression and integrated chunking and encryption may be shown separately, but these principles can be combined according to example embodiments of the present disclosure to implement a scheme that can implement integrated data chunking, compression, and encryption.

[0253] The principles disclosed herein can be applied not only to integrating data chunking into data compression and / or data encryption operations, but also to any type of operation that can scan data, including, for example, filtering, erasure coding, searching, machine learning, graph processing, etc. For example, in Fig. 20 and Fig.25In the illustrated embodiment, the data chunking and compression logic 2090 and / or the data chunking and encryption logic 25101 may be adapted to integrate scanning for chunking with scanning for any other operation. Thus, in some embodiments, a method may include scanning input data, performing operations based on the scan using the input data to generate processed data, finding delimiters in the input data based on the scan, and generating data chunks using the processed data based on the position of the delimiter in the input data. In some embodiments, the data may include an input data stream, and the operation may be a stream-based operation. In some embodiments, the operation may be a block-based operation. In some embodiments, the system may include a host, the host including host logic configured to perform a scan operation on the input data, perform a processing operation based on the scan operation to generate processed data using the input data, find delimiters in the input data based on the scan operation, and generate data chunks using the processed data based on the delimiter. In some embodiments, the system may also include a device configured to receive data chunks from the host, the device including device logic configured to recover the data chunks to generate recovered data chunks and perform operations on the recovered data chunks.

[0254] Fig.26 An example embodiment of a host device for a storage scheme with data block modification according to an example embodiment of the present disclosure is shown. Fig.26 The host 2600 shown can be used to implement any host functionality disclosed herein. The host 2600 can be implemented with any component or combination of components, such as one or more clients, one or more object storage servers, one or more storage nodes, etc. or a combination thereof.

[0255] Fig.26 The illustrated host device 2600 may include a processor 2602 , which may include a memory controller 2604 , a system memory 2606 , host control logic 2608 , and / or a communication interface 2610 . Fig.26 Any or all of the components shown may communicate via one or more system buses 2612. In some embodiments, Fig.26 One or more of the components shown may be implemented using other components. For example, in some embodiments, host control logic 2608 may be implemented by processor 2602 executing instructions stored in system memory 2606 or other memory.

[0256] The host control logic 2608 may include and / or implement any host functionality disclosed herein, including data blocking logic 426, 526 and / or 826, data modification logic 427 and / or 527, compression logic 846, encryption logic 847, erasure coding logic 848, data selection logic 849, cluster management logic 850, node and storage device management logic 851, processing unit 852, content-aware data blocking and compression logic 2090, content-aware data blocking and decryption logic 25101, and the like.

[0257] Fig. 27 An example embodiment of a storage device with data block modification according to an example embodiment of the present disclosure is shown. Fig. 27 The storage device 2700 shown may be used to implement any storage device functionality disclosed herein. The storage device 2700 may include a device controller 2702 , a media conversion layer 2704 , a storage medium 2706 , a computing resource 2708 , a processing control logic 2716 , and a communication interface 2710 . Fig. 27 The components shown may communicate via one or more device buses 2712. In some embodiments where flash memory may be used for some or all of storage media 2706, media translation layer 2704 may be implemented in part or in whole as a flash translation layer (FTL).

[0258] In some embodiments, the processing control logic 2716 may be used to implement any storage device functionality disclosed herein, including data recovery logic 428 and / or 528, processing elements 429, 529 and / or 1429, indication logic 531, data selection engine 853, decryption logic 854, decompression logic 855, data selection logic 856, key value logic 857, decompression logic 2028, one or more processing elements 2029, decryption logic 2528, one or more processing elements 2529, and the like.

[0259] As described above, any functionality described herein, including any host (e.g., client, storage server, storage node, etc.) functionality, storage device functionality, etc. described herein, can be implemented in hardware, software, or any combination thereof, including combinatorial logic, sequential logic, one or more timers, counters, registers, state machines, volatile memory such as DRAM and / or SRAM, non-volatile memory, and / or any combination thereof, CPLDs, FPGAs, ASICs that execute instructions stored in any type of memory, CPUs including CISC processors such as x86 processors and / or RISC processors such as ARM processors, GPUs, NPUs, etc. In some embodiments, one or more components may be implemented as a system on a chip (SOC).

[0260] Fig.28An embodiment of a data compression method according to an example embodiment of the present disclosure is shown. The method may start at operation 2802. At operation 2804, the method may scan input data. In some embodiments, the input data may be received as an input data stream. At operation 2806, the method may perform a compression operation based on the scan to generate compressed data using the input data. At operation 2808, the method may find a delimiter in the input data based on the scan. For example, the input data may include one or more records, and the delimiter may indicate a boundary of a record. At operation 2810, the method may generate data shares using the compressed data based on the position of the delimiter in the input data. The method may end at operation 2812.

[0261] Fig.29 An embodiment of a data encryption method according to an example embodiment of the present disclosure is shown. The method may begin at operation 2902. At operation 2904, the method may scan input data. In some embodiments, the input data may be received as an input data stream. At operation 2906, the method may perform an encryption operation based on the scan to generate encrypted data using the input data. At operation 2908, the method may find a delimiter in the input data based on the scan. For example, the input data may include one or more records, and the delimiter may indicate a boundary of a record. At operation 2910, the method may generate a data share using the encrypted data based on the position of the delimiter in the input data. The method may end at operation 2912.

[0262] Fig.30 An embodiment of a method for dividing data into one or more data portions according to an example embodiment of the present disclosure is shown. The method may begin at operation 3002. At operation 3004, the method may scan input data. In some embodiments, the input data may be received as an input data stream. At operation 3006, the method may perform an operation on the input data based on the scan to generate processed data. In some embodiments, the operation may be any operation that may scan data, such as compression, encryption, filtering, erasure coding, searching, machine learning, chart processing, etc. At operation 3008, the method may search for delimiters in the input data based on the scan. For example, the input data may include one or more records, and the delimiter may indicate the boundary of the record. At operation 3010, the method may generate data shares using the processed data based on the position of the delimiter in the input data. The method may end at operation 3012.

[0263] Fig.28 , Fig.29 and Fig.30The embodiments shown in and all other embodiments described herein are example operations and / or components. In some embodiments, some operations and / or components may be omitted, and / or other operations and / or components may be included. In addition, in some embodiments, the temporal and / or spatial order of operations and / or components may vary. Although some components and / or operations may be shown as separate components, in some embodiments, some components and / or operations shown separately may be integrated into a single component and / or operation, and / or some components and / or operations shown as a single component and / or operation may be implemented with multiple components and / or operations.

[0264] Some embodiments disclosed above have been described in the context of various implementation details, but the principles of the present disclosure are not limited to these or any other specific details. For example, some functions have been described as being implemented by certain components, but in other embodiments, the functions may be distributed between different systems and components in different locations and with various user interfaces. Certain embodiments are described as having specific processes, operations, etc., but these terms also include embodiments in which specific processes, operations, etc. can be implemented with multiple processes, operations, etc., or embodiments in which multiple processes, operations, etc. can be integrated into a single process, step, etc. References to components or elements may refer only to a portion of the component or element. For example, references to blocks may refer to the entire block or one or more sub-blocks. Terms such as "first" and "second" used in the present disclosure and claims may be used only to distinguish the things they modify, and may not indicate any spatial or temporal order unless otherwise apparent from the context. In some embodiments, references to things may refer to at least a portion of the thing, for example, "based on" may refer to "based at least in part on", etc. References to the first element do not imply the presence of the second element. The principles disclosed herein have independent utility and can be implemented separately, and not every embodiment can utilize every principle. However, these principles may also be embodied in various combinations, some of which may amplify the benefits of the individual principles in a synergistic manner.

[0265] According to the inventive principle disclosed in this patent, the various details and embodiments described above can be combined to produce additional embodiments. Since the inventive principle disclosed in this patent can be modified in arrangement and detail without departing from the inventive concept, such changes and modifications are considered to fall within the scope of the appended claims.

Claims

1. A method for data encryption, the method comprising: Receive input data; reading at least a portion of the input data; based on the reading, searching for a delimiter in a portion of the input data; generating data shares using the portion of the input data based on a location of the delimiter in the portion of the input data; and encrypting said data share, in: The input data includes records; The delimiter indicates a boundary of the record; and The data share includes the record. The method according to claim 1 , wherein the position of the delimiter is in a portion of the input data.

3. The method according to claim 2, wherein: Generating the data shares includes generating the data shares based on a subset of the portion of the input data.

4. The method according to claim 1, wherein: The portion of the input data is a first portion of the input data; and The position of the delimiter is in the second portion of the input data.

5. The method according to claim 4, wherein: Generating the data shares includes generating the data shares based on a first portion of the input data and a second portion of the input data. The method of claim 1 , further comprising modifying a size of the data share based on a location of the delimiter.

7. The method according to claim 6, wherein: Modifying the size of the data share includes extending the size of the data share.

8. A method for data encryption, the method comprising: The part that reads the input data; Based on the reading, performing a cryptographic operation to generate encrypted data using a portion of the input data; based on the reading, searching for a delimiter in a portion of the input data; and generating data shares using the encrypted data based on the position of the delimiter in the portion of the input data, in, The input data includes records; The delimiter indicates a boundary of the record; and The data share includes the record.

9. The method according to claim 8, wherein: The generating includes generating the data shares based on a default share size.

10. The method according to claim 8, wherein: The generating includes modifying a size of the data share based on a position of the delimiter in the input data.

11. The method according to claim 8, wherein: The cryptographic operation comprises a block-based cryptographic operation; and The executing includes performing the cryptographic operation on a first block of the input data to generate a first block of encrypted data.

12. The method according to claim 11, wherein: The delimiter is located in a second block of input data, and the method further includes modifying a size of the data share based on a location of the delimiter.

13. The method of claim 12, further comprising modifying the size of the data share based on the size of the second block of the input data.

14. The method according to claim 13, wherein: The modifying includes padding the data share based on a size of a second block of the input data.

15. The method according to claim 14, wherein: The size of the second block of input data is based on a key length for the encryption operation.

16. A method for partitioning data, the method comprising: Read input data; performing a compression operation using a portion of the input data based on the reading to generate processed data; based on the reading, searching for a delimiter in a portion of the input data; and generating data shares using the processed data based on the position of the delimiter in the portion of the input data, in, The input data includes records; The delimiter indicates a boundary of the record; and The data share includes the record.

17. The method of claim 16, wherein: The input data comprises an input data stream; and The compression operation includes a stream-based operation.

Images