Computational Data Storage System

Abstract

In one embodiment, a system includes a host processor and a storage system including one or more storage devices, each including a non-volatile memory and a computational offload controller, wherein the non-volatile memory stores data, and the computational offload controller performs computational tasks on the data based on computational offload commands from the host processor.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This patent application claims the benefit of the filing date of U.S. Provisional Patent Application No. 62 / 817,510, filed March 12, 2019, and entitled “COMPUTATIONAL DATA STORAGE SYSTEMS,” the contents of which are expressly incorporated herein by reference.

[0002] The present disclosure relates generally to the field of computer data storage, and more specifically, but not exclusively, to data storage solutions implemented with computing capabilities. [Background technology]

[0003] In a computing system, data may be stored in block-based storage, such as non-volatile memory (NVM) in a solid-state drive (SSD), locally or across a network. For example, the NVM may be NAND flash memory or any other suitable form of stable persistent storage. As SSDs increase in capacity and internal speed, the NVM is typically limited by the speed of the input / output (I / O) controller to which it is connected and the available bandwidth on the local bus and / or network link. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of a computer system including a host and a block storage device according to various embodiments.

[0005] [Diagram 2] FIG. 2 is a flowchart diagram of a technique for offloading computation to a block storage device, according to various embodiments.

[0006] [Diagram 3]FIG. 1 is a flow diagram of a technique for performing offloaded computing with a block storage device, according to various embodiments.

[0007] [Figure 4] FIG. 1 illustrates a schematic diagram of an exemplary computing device including a block memory device with a non-volatile memory controller as described herein, in accordance with various embodiments.

[0008] [Diagram 5] FIG. 1 illustrates an exemplary storage medium having instructions configured to enable an apparatus to implement various aspects of the present disclosure, according to various embodiments.

[0009] [Figure 6] FIG. 1 illustrates the high-level architecture and data flow of a compute-in-storage stack using block-based storage.

[0010] [Figure 7] FIG. 13 illustrates an example of a computation descriptor for a word count operation.

[0011] [Figure 8] FIG. 1 illustrates the read throughput when using a conventional path and an offload path in this compute-in-storage solution.

[0012] [Figure 9] FIG. 1 illustrates conventional access throughput for checksums.

[0013] [Figure 10] FIG. 13 illustrates checksum offload throughput.

[0014] [Figure 11] FIG. 1 illustrates a typical distributed storage system that stores data using three-way replication.

[0015] [Figure 12] FIG. 1 illustrates an example of searching three-way replicated files in parallel using multiple compute-in storage nodes.

[0016] [Figure 13] FIG. 13 is a diagram showing an example of input parameters that appear at the boundaries of adjacent chunks in multiple replicated storage nodes.

[0017] [Figure 14] A diagram showing an end-to-end parallel computation offloading architecture.

[0018] [Figure 15] FIG. 13 is a flowchart illustrating a parallel computation offload operation in a host process and an SDS node process.

[0019] [Figure 16] FIG. 1 illustrates a flowchart for performing computational tasks on data stored using replicated data storage.

[0020] [Figure 17] FIG. 13 illustrates an example of word count residuals.

[0021] [Figure 18] FIG. 13 is a diagram showing an example of a data layout of two stripes, each with four chunks.

[0022] [Figure 19] FIG. 13 is a diagram showing an example of a data layout and an example of data with 2 stripes, 4 chunks per stripe, block size 12, and sub-block size 3.

[0023] [Figure 20] FIG. 13 shows an example of reconstructing and returning ambiguous residual data.

[0024] [Figure 21]FIG. 1 illustrates an example of identifying residuals in semi-structured data.

[0025] [Figure 22] FIG. 13 is a diagram showing an example of a residual when a character string of length 2 is searched for.

[0026] [Diagram 23] FIG. 13 illustrates an example of a boundary list with residual return.

[0027] [Figure 24] FIG. 2 illustrates a flowchart for performing computational tasks on data objects stored in a striped storage system.

[0028] [Diagram 25] FIG. 1 illustrates an example of a logic flow for erasure coding construction and reconstruction.

[0029] [Figure 26] FIG. 1 illustrates an example of compute-in-storage in a distributed software-defined storage system.

[0030] [Figure 27] FIG. 1 is a diagram illustrating an example of a Ceph OSD configuration in which the information subject to OSD is expanded.

[0031] [Figure 28] FIG. 1 illustrates an example of computation offload to degraded mode storage.

[0032] [Figure 29] FIG. 1 illustrates a flowchart for performing computational tasks on data objects stored in an erasure coded storage system.

[0033] [Diagram 30] FIG. 2 illustrates an example of the overall architecture and data flow of a chained compute-in-storage operation.

[0034] [Diagram 31] FIG. 2 illustrates a flowchart for executing chained computational tasks on data objects stored in a storage system.

[0035] [Diagram 32] FIG. 2 illustrates an example of a striping object with multiple data elements that cross chunk or sub-block boundaries.

[0036] [Diagram 33] FIG. 2 illustrates an example of a striping object with multiple data elements aligned within chunk or sub-block boundaries using padding.

[0037] [Diagram 34] FIG. 2 illustrates an example of a pre-allocated circular buffer.

[0038] [Diagram 35] FIG. 2 shows a flowchart for performing visual computation tasks on image objects stored in a storage system.

[0039] [Diagram 36] FIG. 1 illustrates an example of a computational architecture for aging data using compute-in storage.

[0040] [Figure 37] FIG. 13 illustrates an example of a computational descriptor that compresses data and places it in a new location.

[0041] [Figure 38] FIG. 1 illustrates an example of a computational descriptor that ages data to a pre-allocated location on a different target machine.

[0042] [Figure 39]FIG. 1 illustrates an example of a computational architecture for aging data to different targets using compute-in storage.

[0043] [Diagram 40] FIG. 13 illustrates an example of a computational descriptor for converting duplicate data into erasure encoded data.

[0044] [Diagram 41] FIG. 13 illustrates an example of a computational descriptor for aging data to a third party storage service.

[0045] [Diagram 42] FIG. 2 illustrates a flowchart for performing a data aging calculation task on a data object stored in a storage system.

[0046] [Diagram 43] FIG. 1 illustrates an example of architecture and data flow for hint-based end of life prediction of NAND storage.

[0047] [Diagram 44] FIG. 1 illustrates an example embodiment of end-to-end I / O classification and compute offload descriptor mapping for NAND-based data lifespan prediction.

[0048] [Diagram 45] FIG. 13 is a flowchart showing how data is stored in a NAND storage device using data lifespan prediction.

[0049] [Diagram 46] FIG. 1 illustrates an example process flow for cache eviction and allocation tuning.

[0050] [Figure 47] FIG. 1 illustrates a flow chart for optimizing and configuring a cache in a storage system.

[0051] [Figure 48] FIG. 1 illustrates a flowchart of an example embodiment of offloading CTC computation using Compute-in-Storage.

[0052] [Figure 49] FIG. 2 shows a flowchart for performing CRC verification on data objects stored in a storage system.

[0053] [Figure 50] FIG. 1 is a diagram showing an overview of an edge cloud configuration for edge computing.

[0054] [Figure 51] FIG. 1 illustrates the deployment and orchestration of a virtual edge configuration in an edge computing system operated among multiple edge nodes and multiple tenants.

[0055] [Figure 52] FIG. 1 illustrates a vehicular compute and communications use case including mobile access to applications in an edge compute system.

[0056] [Figure 53] FIG. 1 illustrates a block diagram of a MEC (Multi-Access Edge Computing) system architecture.

[0057] [Figure 54] FIG. 1 illustrates an overview of a hierarchy of distributed computing deployed in an edge computing environment.

[0058] [Figure 55] FIG. 1 is a system diagram for implementing a page table for an edge service.

[0059] [Figure 56] FIG. 13 illustrates an edge service page table.

[0060] [Figure 57]FIG. 1 is a system diagram of an architecture for aggregating data at edge devices.

[0061] [Figure 58A] FIG. 2 illustrates an overview of exemplary components of a compute node.

[0062] [Figure 58B] FIG. 2 illustrates a further overview of exemplary components within a computing device.

[0063] [Figure 58C] FIG. 1 illustrates an overview of exemplary components in a configurable server rack. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0064] This patent application claims the benefit of the filing date of U.S. Provisional Patent Application No. 62 / 817,510, filed March 12, 2019, and entitled “COMPUTATIONAL DATA STORAGE SYSTEMS,” the contents of which are expressly incorporated herein by reference.

[0065] In the following disclosure, a number of different embodiments or examples are provided to implement different features of the present disclosure. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. Furthermore, the present disclosure may repeat reference numbers and / or reference characters in various examples. This repetition is for the purpose of brevity and clarity, and the repetition itself does not dictate a relationship between the various embodiments and / or configurations discussed. Different embodiments may have different advantages, and a particular advantage is not necessarily required for every embodiment.

[0066] This disclosure presents various embodiments that relate to computational storage or "compute-in-storage," which refers to data storage solutions that implement computing functionality.

[0067] [Computation offloading with block storage] FIG. 1 is a block diagram of a computer system 100 that may include a host 102 and a block storage device 104 (e.g., a block-based storage device such as an SSD, a block-based storage server, or any other suitable block-based storage device) according to various embodiments. In some embodiments, the host 102 may include a processor 106 coupled with a memory 108. In various embodiments, a computational process 110 may be executed on the host 102 (e.g., by the processor 106 in the memory 108). In some embodiments, the computational process 110 may be an application, a storage middleware, a software storage stack, an operating system, or any other suitable computational process. In various embodiments, the host 102 may further include a computational offload device 112 that may include a client offload logic 114 and an initiator 116. In some embodiments, the block storage device 104 may be referred to as a storage object or a computable storage object. In some embodiments, the client offload logic 114 is referred to as a client offload device.

[0068] In some embodiments, the computing process 110 may send a computation offload request to the computation offload device 112. In various embodiments, the computation offload request may specify a high-level object (e.g., a file) and a desired operation (e.g., a hash function such as an MD5 operation). In some embodiments, the client offload logic 114 may build a block-based computational descriptor 130 based at least in part on the request and package the block-based computational descriptor 130 into a computation offload command 132. In various embodiments, the computation offload command 132 may be a vendor-specific command that may include block-based metadata (e.g., as part of the block-based computational descriptor 130). In some embodiments, the client offload logic 114 may generate a virtual input object 134 based at least in part on the high-level object specified by the computation offload request. In some embodiments, the client offload logic 114 may determine a list of one or more blocks corresponding to where the high-level object is stored in block-based storage (e.g., NVM) to generate the virtual input object 134.

[0069] In various embodiments, the block-based computational descriptor 130 may describe the storage blocks to be input and / or output for the computation (e.g., those mapped by the virtual object), the function 138 to be performed (e.g., the requested computation operation identified by a computation type identifier or an operation code), and any additional arguments 140 (e.g., a search string) to the function 138. In various embodiments, the additional arguments 140 may also be referred to as parameters. In some embodiments, the computational offload device 112 may include a client offload library 115 that may be used by the client offload logic 114 in creating the block-based computational descriptor 130. In some embodiments, the client offload library 115 may not be present and / or some or all aspects of the client offload library 115 may be included in the client offload logic 114 (e.g., in an ASIC). In various embodiments, the client offload logic 114 may create virtual input objects 134 and / or virtual output objects 136 (e.g., lists of block extents and object lengths) and may assign operation codes for the desired operations to be performed with these virtual objects. In some embodiments, a compute offload command 132 with a block-based compute descriptor 130 may contain all of the necessary information to schedule a computation on a virtual object (e.g., a virtual input object 134) in the block storage device 104. In various embodiments, the block-based compute descriptor 130 may describe a block-based computation operation in a protocol-agnostic manner that may work for any block-based storage device or system. In some embodiments, the virtual input object 134 may include a first set of metadata that maps the virtual input object 134 to a real input object (e.g., a file).In various embodiments, the first set of metadata may include a size of the real input object, a list of blocks that make up the real input object, and / or any other metadata that describes the real input object. In some embodiments, the virtual output object 136 may include a second set of metadata that maps the virtual output object 136 to a real output object.

[0070] Various embodiments may perform generic file-based computations on block-based storage and / or include all execution context in a single I / O command (e.g., computation offload command 132). This may provide a performance advantage over traditional approaches that require multiple round trips (e.g., communication) between the host and the target to initiate target-side computations and / or have scheduling overhead that scales (e.g., linearly) with the number of blocks in the file. By including all execution context in a single I / O command, various embodiments may provide an advantage over traditional approaches that use programmable filters that persist across READ operations and / or require separate initialization and finalization commands (e.g., introducing state tracking overhead to SSD operations). Some embodiments may not require the introduction of an object-based file system anywhere on the host, which may reduce complexity compared to traditional approaches. Some embodiments may provide a generic solution suitable for use with any file system and may work with an object-based storage stack, unlike traditional approaches that require applications to access block storage directly and / or are not suitable for use with a file system.

[0071] In some embodiments, the initiator 116 may communicate a compute offload command 132, including a block-based compute descriptor 130, to the block storage device 104 over the link 118. In some embodiments, the link 118 may be a transport fabric, such as an iSCSI (Internet Small Computer System Interface), an NVMeOF (NVM over Fabric Express) interface, or any other suitable transport fabric. In other embodiments, the link 118 may be a local bus interface, such as PCIe (Peripheral Component Interconnect Express), or any other suitable interface.

[0072] In various embodiments, the block storage device 104 may include an NVM 120 and a computational offload controller 122. In some embodiments, the computational offload controller 122 may be or be part of an NVM controller, an SSD controller, a storage server controller, or any other suitable block-based storage controller. While the NVM 120 is shown as a single element for clarity, it should be understood that in various embodiments, multiple NVMs 120 may be present in the block storage device 104 and / or be at least partially controlled by the computational offload controller 122. In some embodiments, the computational offload controller 122 may include parsing logic 124 and computation logic 126. In various embodiments, the parsing logic 124 may parse computational offload commands (e.g., computational offload commands 132) and / or computational descriptors (e.g., block-based computational descriptors 130) received from the host 102. In some embodiments, the parsing logic 124 identifies a compute descriptor (e.g., block-based compute descriptor 130) packaged in a compute offload command (e.g., compute offload command 132) and parses the identified compute descriptor to identify a virtual input object (e.g., virtual input object 134), a virtual output object (e.g., virtual output object 136), a requested compute operation (e.g., function 138), and / or other parameters (e.g., a search string specified in additional arguments 140). In various embodiments, the compute logic 126 performs the requested compute operation. In some embodiments, the compute logic 126 may perform the requested compute operation 138 on the virtual input object 134 and store the results of the requested compute operation in the virtual output object 136. In some embodiments, the compute offload controller 122 and / or the compute logic 126 may be implemented using hardware accelerators designed to perform specific compute operations on data stored in the non-volatile memory 120.

[0073] In some embodiments, while an offloaded computation is being performed, one or more standard operations (e.g., read and write operations) of NVM 120 may continue to operate normally. In some embodiments, computation offload controller 122 may include a target offload library 127 that may be used by parsing logic 124 in parsing computation offload commands and / or computation descriptors and by computation logic 126 to perform the requested computation. In some embodiments, target offload library 127 may not be present and / or some or all aspects of target offload library 127 may be included in parsing logic 124 and / or computation logic 126 (e.g., in an ASIC). In some embodiments, if one or more expected items are not included in a descriptor (e.g., a virtual output object), a default value may be used or a default action may be performed, if possible. In various embodiments, creating virtual objects in a block storage system and performing computations on the virtual objects may avoid problems associated with traditional approaches to adding complex object-based devices and object-based file systems. In some embodiments, the parsing logic 124 is referred to as a parser and the computation logic 126 is referred to as an offload executor.

[0074] In various embodiments, the virtual input object 134 may include a first list of one or more blocks. In some embodiments, the first list of one or more blocks may include a list of starting addresses and a corresponding list of block lengths to form a first set of block extents. In various embodiments, the virtual output object 136 may include a second list of one or more blocks. In some embodiments, the second list of one or more blocks may include a list of starting addresses and a corresponding list of block lengths to form a second set of block extents. In other embodiments, the first and / or second set of block extents may be specified by a list of starting addresses and a list of ending addresses and / or may include a total virtual object length (the virtual input object length or the virtual output object length, respectively). In some embodiments, the requested computation operation 138 may be a function (e.g., compression, hashing, searching, image resizing, checksum calculation, or any other suitable function) that may be applied to the first list of one or more blocks and written to the second list of one or more blocks. In some embodiments, the blocks associated with the virtual input object 134 and / or the virtual output object 136 may be sectors. In some embodiments, the starting address may be a logical block address (LBA), the one or more blocks of the first and second lists may be otherwise identified by an LBA, or the first and / or second lists of one or more blocks may be identified in any other suitable manner. In various embodiments, the virtual input object 134 may specify block locations in NVM 120 where the file data is stored, and / or the virtual output object 136 may specify block locations in NVM 120 where the results are written. In some embodiments, the virtual output object 136 may specify that the results are to be returned to the host 102.

[0075] In various embodiments, the parsing logic 124, the computation logic 126, and / or other functions of the computation offload controller 122 may be executed by one or more processors or central processing units (CPUs), one or more field programmable gate arrays (FPGAs), one or more application specific integrated circuits (ASICs), an intelligent storage acceleration library (ISA-L), a data streaming architecture, and / or any other suitable combination of hardware and / or software not shown for clarity. In some embodiments, the computation offload controller 122 may include one or more buffers 128, which in various embodiments may include input buffers, output buffers, and / or input / output buffers. In some embodiments, one or more components of the computation offload controller 122 (e.g., the computation logic 126 and / or the parsing logic 124) may use the buffers 128 in read and / or write operations to the NVM 120.

[0076] In some embodiments, block storage 104 may be an SSD, which in various embodiments may be coupled to host 102 via a local bus, such as PCIe, or coupled to host 102 via a network. In some embodiments, block storage 104 may be a storage server that may be part of a distributed computing environment. In various embodiments, host 102 and / or block storage 104 may include additional elements not shown for clarity (e.g., block storage 104 may include one or more processors and system memory).

[0077] In various embodiments, NVM 120 may be a memory whose state is determinable even when power to the device is interrupted. In some embodiments, NVM 120 may include a block-addressable mode memory device, such as NAND or NOR technology, and more specifically, multiple threshold level NAND flash memory (e.g., single-level cell ("SLC"), multi-level cell ("MLC"), quad-level cell ("QLC"), tri-level cell ("TLC"), or other NAND). In some embodiments, NVM 120 may include byte-addressable, write-in-place, three-dimensional cross-point memory devices or other byte-addressable, write-in-place NVM devices such as single or multi-level phase change memory (PCM) or switched phase change memory (PCMS), NVM devices using chalcogenide phase change materials (e.g., chalcogenide glasses), resistive memory including metal oxide-based, oxygen vacancy-based, and conductive bridge random access memory (CB-RAM), nanowire memory, ferroelectric transistor random access memory (FeTRAM), magnetoresistive random access memory (MRAM) incorporating memristor technology, spin-transfer torque (STT)-MRAM, spintronic magnetic junction memory-based devices, magnetic tunnel junction (MTJ)-based devices, DW (domain wall) and SOT (spin-orbit transformation)-based devices, thyristor-based memory devices, any combination of the above, or other suitable memories.

[0078] In various embodiments, offloaded computational operations (eg, calculating checksums, detecting bit rot, etc.) take less time to complete and / or reduce I / O traffic as compared to legacy approaches.

[0079] 2 is a flow chart diagram of a technique 200 for offloading computational processing to a block storage device, according to various embodiments. In some embodiments, some or all of the technique 200 may be performed by components shown and / or described with respect to computer system 100 of FIG. 1, or portions thereof (e.g., computation offload unit 112), computing device 400 of FIG. 4, or portions thereof (e.g., computation offload unit 450), or any other components shown and / or described herein with respect to other figures.

[0080] In some embodiments, at block 202, technique 200 may include receiving a request from a computing process (e.g., receiving a request by computation offload device 112 from computing process 110). In various embodiments, the request may include a higher-level object (e.g., a file), a requested operation or calculation, and one or more additional parameters (e.g., a search string). However, it should be understood that in other embodiments, the request may include any other suitable parameters.

[0081] In various embodiments, at block 204, technique 200 includes constructing a block-based computational descriptor (e.g., block-based computational descriptor 130) based at least in part on the request. In some embodiments, constructing the block-based computational descriptor may include constructing an extent map (e.g., a block list of the virtual input object) based at least in part on a higher-level object (e.g., a file) included in the request. In some embodiments, the extent map may include a list of LBAs.

[0082] In some embodiments, at block 206, technique 200 includes sending a block-based computational descriptor to a block-based storage device (e.g., block storage device 104 or block storage device 452) using a computational offload command. In some embodiments, sending the block-based computational descriptor to the block-based storage device may include loading the block-based computational descriptor into a payload (e.g., the computational offload command). In various embodiments, the block-based storage device may be an NVM storage device.

[0083] In some embodiments, the computational offload command sent in block 206 may be a SCSI command that is transferred over the network using the iSCSI transport protocol. In some embodiments, the SCSI command that is transferred over the network using the iSCSI transport protocol may be referred to as an iSCSI command. In some embodiments, the computational offload command sent in block 206 may be an iSCSI command that may use an operation code (opcode) specified as (0x99). In some embodiments, the (0x99) command may be defined as a bidirectional command that includes an output buffer and an input buffer. In some embodiments, the output buffer of the (0x99) command may store a computational descriptor, and the input buffer of the (0x99) command may store a result that is executed according to the computation described in the computational descriptor. In some embodiments, the (0x99) command may be defined as a vendor-specific command and / or may be referred to as an EXEC command. In other embodiments, the computational offload command may be a SCSI command defined in a manner similar to the iSCSI commands described above, but may be transferred directly to the attached device (e.g., via a local bus, such as a PCIe bus). It should be understood that the (0x99) command is mentioned for illustrative purposes, and any suitable opcode designation or other computation offload command identifier may be used in various embodiments.

[0084] In some embodiments, the compute offload command sent in block 206 may include one or more NVMe commands. In some embodiments, the compute offload command may be a fused NVMe command that includes two opcodes. In some embodiments, the fused NVMe command may include a first opcode that may be used to transfer a compute descriptor from the host to the block-based storage device and a second opcode that may then be used to transfer the results from the block-based storage device back to the host. In this manner, the fused NVMe command may be a virtual bidirectional command by fusing two unidirectional commands. In some embodiments, the first opcode may be a vendor-specific opcode designated as (0x99) and / or may be referred to as an EXEC_WRITE command. In some embodiments, the second opcode may be a vendor-specific opcode designated as (0x9a) and / or may be referred to as an EXEC_READ command. In some embodiments, the EXEC_WRITE command may be equivalent to the first phase of the iSCSI bidirectional EXEC command described above (e.g., includes a computational descriptor), and / or the EXEC_READ command may be equivalent to the second phase of the iSCSI bidirectional EXEC command described above (e.g., returns the result of the operation). In some embodiments, the fused NVMe command may be sent over the network using the NVMeOF transfer protocol. In some embodiments, an NVMe command transferred over the network using the NVMeOF transfer protocol may be referred to as an NVMeOF command. In some embodiments, the fused NVMe command may be transferred directly to the attached device (e.g., via a local bus such as a PCIe bus). In some embodiments, the iSCSI or SCSI computational offload command (e.g., EXEC) may be converted to the fused NVMe command described above before sending it to the NVM storage device. It should be understood that in other embodiments, any other suitable computational offload command may be used.It should be understood that the vendor-specific opcodes (0x99) and (0x9a) are provided for illustrative purposes, and that any suitable opcode designation(s) or other compute offload command identifier(s) may be used in various embodiments.

[0085] In some embodiments, at block 208, technique 200 may include receiving results from a block-based storage device in response to the block-based computation descriptor. In various embodiments, at block 210, technique 200 may include performing other operations.

[0086] 3 is a flow diagram of a technique 300 for performing offloaded computational processing using block storage, according to various embodiments. In some embodiments, some or all of the technique 300 may be performed by components shown and / or described with respect to computer system 100 of FIG. 1 or portions thereof (e.g., block storage 104 and / or computation offload controller 122), computer device 400 of FIG. 4 or portions thereof (e.g., block storage 452 and / or computation offload controller 454), or other components shown or described herein with respect to any other figures.

[0087] In some embodiments, at block 302, technique 300 may include receiving a block-based computational descriptor (e.g., receiving block-based computational descriptor 130 from computational offload device 112 at block storage 104 or receiving block-based computational descriptor from computational offload device 450 at block storage 452). In various embodiments, at block 304, technique 300 may include parsing the block-based computational descriptor (e.g., with parsing logic 124). In some embodiments, at block 306, technique 300 may include creating a context. In various embodiments, parsing logic 124 and / or any other suitable component of computational offload controller 122 or computational offload controller 454 may create the context. In some embodiments, the context may include one or more of the following: an operation to perform (e.g., a text search, etc.), one or more arguments for the operation (e.g., a search string, etc.), whether the operation expects data to arrive across multiple calls or requires all data to be input as a single buffer, and / or additional state information specific to the operation (e.g., the current state of a checksum calculation on the chunked input, etc.). In some embodiments, whether the operation can expect data to arrive across multiple calls is opaque to the calling application (e.g., computing process 110), but may be relevant for performing operations such as those that require reading multiple block extents for a particular virtual object. In some embodiments, the context may be a computation context that can provide temporary space for the input and results of the operation.

[0088] In various embodiments, at block 308, technique 300 may include reading data into one or more buffers (e.g., input buffers of buffer 128). In some embodiments, reading data into one or more buffers may include performing a check to determine whether sufficient data has been read into the one or more buffers for performance of the requested operation before proceeding to decision block 310. In some embodiments, at decision block 310, technique 300 may include determining whether the operation code from the block-based computation descriptor is in a list of available operations. If at decision block 310, it is determined that the operation code is not in the list of available operations, technique 300 may include returning an error at block 312. If at decision block 310, it is determined that the operation code is in the list of available operations, technique 300 may include performing the operation based at least in part on the operation code at block 314. In some embodiments (e.g., where operations may be performed on a subset of the data rather than the entire data set), technique 300 may include looping the operations performed at blocks 308 and 314 to perform operations on the subset of the virtual input object until the entire virtual input object has been processed.

[0089] In some embodiments, at block 316, technique 300 may include storing a result of the operation performed at block 314. In various embodiments, the result may be stored in a location of a virtual output object and / or returned to a host (e.g., host 102). In some embodiments, returning the result to the host may include copying the result data to a return payload of the compute offload command. In some embodiments, at block 318, technique 300 may include performing other operations. In various embodiments, one or more of the operations performed in technique 300 may be specified in hardware, fixed as a static library, dynamically loaded at run time, or implemented in any suitable combination of hardware and / or software. In some embodiments, one or more of the operations described with respect to technique 300 may be performed in a different order (e.g., determining whether the operation code is in the list at block 310 may be performed before reading the data into one or more buffers at block 308, and returning an error before reading the data into the buffer if the operation code is not in the list).

[0090] FIG. 4 illustrates a block diagram of an example computing device 400 that may be suitable for use with various components of FIG. 1, the technique 200 of FIG. 2, and / or the technique 300 of FIG. 3, in accordance with various embodiments.

[0091] As shown, the computing device 400 may include one or more processors or processor cores 402 and a system memory 404. For purposes of this application, including the claims, the terms "processor" and "multiple processor cores" may be considered synonymous unless the context clearly requires otherwise. The processor 402 may include any type of processor, such as a central processing unit (CPU), microprocessor, etc. The processor 402 may be implemented as an integrated circuit having multiple cores, e.g., a multi-core microprocessor. In some embodiments, the processor 402 may further include a hardware accelerator in addition to the cores, e.g., a hardware accelerator implemented in an FPGA (field programmable gate array). The computing device 400 may include a mass storage device 406 (a diskette, a hard drive, a non-volatile memory (NVM) (e.g., a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), any other type of suitable NVM, etc.). In general, the system memory 404 and / or the mass storage devices 406 may be any type of temporary and / or persistent storage, including, but not limited to, volatile and non-volatile memory, optical, magnetic and / or solid state mass storage, etc. Volatile memory may include, but is not limited to, static and / or dynamic random access memory (DRAM). Non-volatile memory may include, but is not limited to, electrically erasable programmable read-only memory, phase change memory, resistive memory, etc. In some embodiments, the mass storage device 406 may include the NVM 120 described with respect to FIG. 1.

[0092] Computing device 400 may further include I / O devices 408 (such as a display (e.g., a touch screen display), a keyboard, a cursor control, a remote control, a game controller, an image capture device, etc.), a communications interface 410 (such as a network interface card, a modem, an infrared receiver, a wireless receiver (e.g., Bluetooth), etc.), one or more antennas, and / or any other suitable components.

[0093] The communications interface 410 may include a local area network (LAN) (e.g., Ethernet) and / or a communications chip (not shown for clarity) configured to operate the computing device 400 in accordance with a Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or Long Term Evolution (LTE) network. The communications chip may also be configured to operate in accordance with Enhanced Data GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communications chips may be configured to operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), their derivatives, and any other wireless protocols designated as 3G, 4G, 5G and beyond. The communications interfaces 410 may operate according to other wireless protocols in other embodiments.

[0094] In various embodiments, the computing device 400 may include a block storage 452, which may include a computational offload controller 454 and / or an NVM 456. In some embodiments, the block storage 452 or components thereof may be coupled to other components of the computing device 400. In some embodiments, the block storage 452 may include a different number of components (e.g., the NVM 456 may be located in the mass storage 406) or may include additional components of the computing device 400 (e.g., the processor 402 and / or the memory 404 may be part of the block storage 452). In some embodiments, the computational offload controller 454 may be configured similarly to the computational offload controller 122 described with respect to FIG. 1. In some embodiments, for example, the computational offload controller 454 may include a hardware accelerator designed to perform specific computational operations on data stored in the non-volatile memory 456.

[0095] In various embodiments, the computing device 400 may include a computational offload unit 450. In some embodiments, the computational offload unit 450 may be configured similarly to the computational offload unit 112 described with respect to FIG. 1. In some embodiments, the computing device 400 may include both the computational offload unit 450 and the block storage unit 452 (e.g., as an SSD), and the computational offload unit 450 may send computational offload commands (e.g., NVMe or SCSI) including computational descriptors to the block storage unit 452 via a local bus. In other embodiments, the first computing device 400 may include the computational offload unit 450 and the second computing device 400 may include the block storage unit 452, and the computational offload unit 450 may send computational offload commands (e.g., iSCSI or NVMeOF) to the block storage unit 452 via a network (e.g., via the communication interface 410). In some embodiments, the first computing device 400 and the second computing device 400 may be components of a distributed computing environment, and the second computing device 400 having the block storage device 452 is a storage server that may include compute-in-storage functionality provided by the block storage device 452.

[0096] The elements of the computing device 400 described above may be coupled together via a system bus 412, which may represent one or more buses. In the case of multiple buses, they may be bridged by one or more bus bridges (not shown). Each of these elements may perform its conventional functions as known in the art. In particular, the system memory 404 and mass storage device 406 may be employed to store working and permanent copies of programming instructions for the operation of various components of the computing device 400, including, but not limited to, the operating system of the computing device 400, one or more applications, operations related to the computing device 400, operations related to the block storage device 452, and / or operations related to the computational offload unit 450, collectively denoted as computational logic 422. The various elements may be implemented by assembler instructions supported by the processor(s) 402 or a high-level language that may be compiled into such instructions. In some embodiments, computing device 400 may be implemented as a fixed function ASIC, an FPGA, or any other suitable device, with or without programmability or configuration options.

[0097] A permanent copy of the programming instructions may be placed on mass storage device 406 at a factory, or may be located in the field (e.g., from a distributed server (not shown)) on a distribution medium such as a compact disc (CD), or through communications interface 410. That is, one or more distribution media having an implementation of the agent program may be distributed to agents and used to program various computing devices.

[0098] The number, capabilities and / or capacity of elements 408, 410, 412 may vary depending on whether computing device 400 is used as a fixed computing device, such as a set-top box or a desktop computer, or a mobile computing device, such as a tablet computing device, a laptop computer, a game console or a smartphone. These configurations are otherwise known and will not be described in further detail.

[0099] For some embodiments, at least one of the processors 402 may be packaged with computational logic 422 configured to implement aspects of the embodiments described herein to form a system-in-package (SiP) or system-on-chip (SoC).

[0100] In various implementations, computing device 400 may comprise one or more components of a data center, a laptop, a netbook, a notebook, an ultrabook, a smartphone, a tablet, an ultra-mobile PC, or a mobile phone. In some embodiments, computing device 400 may include one or more components of a server. In further implementations, computing device 400 may be any other electronic device that processes data.

[0101] FIG. 5 illustrates an exemplary computer-readable storage medium 502 having instructions configured to perform all or selected operations associated with the computing device 400 described above with respect to FIG. 4, the computer system 100 described with respect to FIG. 1, the computational offload controller 122, and / or the computational offload apparatus 112, the technique 200 described with respect to FIG. 2, and / or the technique 300 of FIG. 3, in accordance with various embodiments.

[0102] As shown, computer readable storage medium 502 may include a number of programming instructions 504. Storage medium 502 may represent a wide variety of non-transitory persistent storage media known in the art, including, but not limited to, flash memory, dynamic random access memory, static random access memory, optical disks, magnetic disks, etc. In response to execution of programming instructions 504, the programming instructions 504 may cause devices that are part or all of computer system 100 and / or computing device 400, such as, for example, computation offload controller 122, computation offload unit 112, and / or other components of computer system 100, to perform various operations, such as, for example, but not limited to, computation offload controller 122, parsing logic 124, computation logic 126, computation offload unit 112, client offload logic 114, initiator 116, block storage 452 and / or computation offload unit 450 of FIG. 4, technique 200 described with respect to FIG. 2, and / or technique 300 of FIG. 3. In alternative embodiments, the programming instructions 504 may be located on computer readable storage media 502. In alternative embodiments, the storage media 502 may be transitory, such as, for example, a signal encoded with the programming instructions 504.

[0103] 4, in one embodiment, at least one of the processors 402 may be packaged together with the computation offload controller 122, the parsing logic 124, the computation logic 126, the computation offload unit 112, the client offload logic 114, the initiator 116, and / or other components of the computer system 100 shown in FIG. 1, a computing device 400 including the block storage 452 and / or the computation offload unit 450 of FIG. 4, a memory having all or a portion of the computation logic 422 configured to perform aspects shown or described with respect to the technique 200 described with respect to FIG. 2, and / or the technique 300 of FIG. 3. In one embodiment, at least one of the processors 402 may be packaged with the computation offload controller 122, the parsing logic 124, the computation logic 126, the computation offload unit 112, the client offload logic 114, the initiator 116, and / or other components of the computer system 100 shown in FIG. 1, the computation device 400 including the block storage 452 and / or the computation offload unit 450 of FIG. 4, the technique 200 described with respect to FIG. 2, and / or the technique 300 of FIG. 3, with all or a portion of the computation logic 422 configured to perform aspects shown or described in relation to the technique 200 described with respect to FIG. 2, and / or the technique 300 of FIG. 3 to form a system-in-package (SiP). In one embodiment, at least one of the processors 402 may be integrated on the same die as the computational offload controller 122, the parsing logic 124, the computational logic 126, the computational offload unit 112, the client offload logic 114, the initiator 116, and / or other components of the computer system 100 shown in FIG. 1, a computing device 400 including the block storage 452 and / or the computational offload unit 450 of FIG. 4, a memory having all or a portion of the computational logic 422 configured to perform aspects shown or described with respect to the technique 200 described with respect to FIG. 2, and / or the technique 300 of FIG. 3.In one embodiment, at least one of the processors 402 may be packaged with the computation offload controller 122, the parsing logic 124, the computation logic 126, the computation offload unit 112, the client offload logic 114, the initiator 116, and / or other components of the computer system 100 shown in FIG. 1, the computation device 400 including the block storage 452 and / or the computation offload unit 450 of FIG. 4, the technique 200 described with respect to FIG. 2, and / or the technique 300 of FIG. 3, with all or a portion of the computation logic 422 configured to perform aspects shown or described with respect to the technique 200 described with respect to FIG. 2, and / or the technique 300 of FIG. 3 to form a system on a chip (SoC).

[0104] [Computational memory using virtual objects] Computational memory has been an elusive goal until now. While it is technically correct to place computation close to storage and minimize data movement, many attempts to do so have failed because storage first needed to evolve from block protocols to more computationally friendly object or key-value protocols. This evolution has not happened yet.

[0105] However, block protocol modifications are not necessarily required to achieve computational storage performance. For example, in the next section, we describe a block-based (i.e. legacy-compatible) computational storage design based on virtual objects. Like real objects (e.g., files), "virtual" objects contain the metadata necessary to process the underlying data, so numerous offloads are possible with existing block storage systems. As an end-to-end example, offloading bitrot detection in an object store can achieve a 99% reduction in network traffic.

[0106] Advances in storage technology have led to extremely fast devices. Solid-state disks use various forms of non-volatile memory, especially NAND and persistent memory. However, I / O speeds have not kept up, and as a result, data transfers between storage and the CPU take time. The available bandwidth for these high-speed devices is managed by an I / O stack that includes a variety of legacy software and hardware. Current efforts to improve the I / O path do not solve the fundamental problem of data taking so long to move.

[0107] Consider a modern storage server. Current SSDs can deliver speeds of around 3.0Gb / s (or 24Gb / s), but commodity Ethernet is limited to 100Gb / s. When building a storage server with around 16-32 drives, this network bottleneck quickly becomes apparent and trade-offs must be made. Unfortunately, this often means that the parallel bandwidth of all the drives in the storage server is not available. The same problem occurs with a single SSD, as the bandwidth of the internal NAND exceeds the bandwidth of the storage controller.

[0108] Computational storage often attempts to solve the data movement problem: by placing computation and data physically close together, expensive I / O can be avoided.

[0109] While various designs have been pursued, none have been sufficiently advantageous to justify a new storage protocol. In particular, there is a near-universal call for new object storage protocols, with respect to previous computational storage approaches. These approaches not only introduce too much disruption to the computer storage industry (e.g., the T10 object-based storage SCSI standard), but also ignore existing large-scale block storage deployments.

[0110] For this reason, in what follows we describe a block-based approach to computational storage that enables a large number of different computational offloads without requiring changes to the block protocol, which is especially beneficial given the growing trend towards decoupling block storage (e.g., iSCSI, NVMeOF, etc.).

[0111] Rather than starting with an object-based solution, we use a block-based solution and add in additional pieces such as metadata. For example, the metadata required for a block storage system to understand a group of unordered sectors can be leveraged to define a "virtual object". After all, an object is nothing more than a collection of blocks, a length, and other metadata such as timestamps and ownership. Such metadata is small and can be shared very easily with the block storage system, allowing the storage to load the object into memory for further processing.

[0112] A virtual object is wrapped in a structure called a "compute descriptor" along with any additional information needed to process the object data (e.g., how to execute it and its arguments). The compute descriptor is sent from the host to the target using a new SCSI or NVMe command, referred to here as the "execute" (EXEC) command. The data returned by the EXEC command is the result of the requested computation.

[0113] The client-side library manages creating virtual objects, creating compute descriptors, issuing EXEC commands, etc. Block READ and block WRITE commands remain unchanged, allowing legacy applications to continue to work with the block storage unchanged.

[0114] This approach can be implemented using both iSCSI and NVMeOF, enabling a variety of possible computational offloads (e.g., word counting, searching, compression, checksumming, video transcoding, image conversion, sorting, merging, etc.). Similar techniques can be applied to other forms of block storage, including disk drives, solid-state drives, disk arrays, and clustered storage systems (e.g., Amazon's Elastic Block Storage).

[0115] Figure 6 shows the high-level architecture and data flow of a compute-in-storage stack with block-based storage. As a simple illustrative example of the architecture in Figure 6, consider a word count application. Word count is an ideal use case for compute-in-storage, with large input data but small output, resulting in maximum traffic reduction. On the host, the application uses the client's Object Aware Storage (OAS) library to create virtual objects and compute descriptors, and sends the command to the initiator for transfer. The transfer sends an EXEC command (a vendor-specific command indicating that the payload is a compute descriptor) to the target. The target then interprets the EXEC command and passes it to the specified offload module (word count in this case) for processing. The word count result is returned as payload to the host via the transfer, and then returned to the calling application as a simple memory buffer.

[0116] The solution is built around a new EXEC command for the block storage stack. The EXEC command can be easily plugged into an existing stack and does not change the underlying transfer protocol at all. The EXEC command contains a virtual object and a computation descriptor. The virtual object is a block and length mapping that describes the computation target object, embedded in the computation descriptor. The computation descriptor tells the target what operation to perform on the virtual object, the function arguments, and how / where to output the result. Figure 7 shows an example computation descriptor for a word count operation. The arguments and output are not needed for this operation so they are NULL and the result can be returned directly to the host.

[0117] The host side of this computation-storage approach mainly involves creating computation descriptors and (potentially) pre-allocating files to store output results, which is mostly abstracted from the underlying application using the OAS library.

[0118] Virtual object creation is handled "under the hood" by combining feemaps and stats. Operations and arguments are defined by simple opcodes and strings in the compute descriptor. Preallocated files can be created for output. Once the compute descriptor has been created and embedded as the payload into the desired transport protocol, it is sent to the initiator as an ioctl (e.g. an input / output control system call).

[0119] This solution can be implemented using either iSCSI or NVMeOF (among other examples), in either case keeping the low level transport protocol unchanged. In the case of iSCSI, the host side uses existing bidirectional commands, while in the case of NVMeOF it can leverage fused commands, such as mixing writes and reads. Writes contain a computation descriptor, and reads return the result of the computation.

[0120] When a target receives an EXEC command and its compute descriptor, a computation context is created that contains state and instructions that are understood by the target's compute library and its associated offload modules, which perform the actual computations on the data specified by the virtual objects.

[0121] Results are either stored directly in the context and then returned to the caller or written to the compute descriptor location. Results smaller than the preallocated location are noted on the return and can be truncated on the host side, but care must be taken with caching to ensure host and target consistency.

[0122] All legacy commands are allowed to pass through as normal, so existing applications can continue to work as normal without any changes.

[0123] To illustrate the performance of this compute-in-storage architecture, we performed a performance evaluation using bit rot detection as an example. Bit rot detection, sometimes called an audit or fixity check, is the process of hashing object data and comparing it to a previously computed hash to detect corruption. Such corruption can have a variety of causes, including software bugs or latent sector errors.

[0124] In traditional object storage, where small storage devices are directly attached, the overhead for bitrot detection is relatively small. However, two trends are making this common operation increasingly painful. First, storage devices are becoming larger and larger, requiring more throughput to effectively catch object corruption. Second, there is a growing movement towards further isolation at the block level, as well as distributed storage in the form of object storage systems like Ceph and Swift, leading to "diskless" deployments of object storage systems. These types of deployments are attractive for applications and environments that require private object storage, but not at a scale that requires a full "bare metal" deployment. However, this trend means that bitrot detection, which was traditionally a purely local operation, now generates network traffic and can compete with the traffic of more useful applications.

[0125] For this evaluation, we used a simple two-box lab setup. The target system has eight Intel® P4600 1.6TB SSDs and dual Xeon Gold 6140s. The host / initiator system has dual Intel® Xeon E52699s. Both systems are running the above compute-in-storage stack with NVMeOF on a modified 4.13.2 kernel of Ubuntu Linux® 16.04. The host and target are connected with a 40GbE link using a Chelsio T580-LP-CR card.

[0126] A custom benchmark utility is used to compare traditional computation, which reads data over the network and performs computations on the host, with offload computation, which computes in-situ on the block storage target and returns the results as an NVMe payload.

[0127] In these experiments, we place 100 16MB files on each SSD and expose them as devices to the underlying NVMeOF compute-in-storage stack. Additionally, all experiments use 48 worker threads. The evaluation involves a single run, pulling data from 1-8 devices and measuring traditional and offloaded throughput for a given operation. All traditional operations are measured with forced SSD reads due to a cold page cache, and offloaded operations reading directly from the SSD(s).

[0128] In the first set of experiments, we focused on read performance (NOP) to understand the performance of the compute-in-storage stack. Figure 8 shows the read throughput using conventional and offload paths for this compute-in-storage solution. The dotted line refers to the theoretical maximum throughput of the network and SSDs. Furthermore, since the offload is purely on the target, there is no gating by the network. As shown in Figure 8, conventional performance is rapidly gated by the 40GbE link, as expected. Offloaded reads scale with the number of SSDs doing the reads, and even with 8 devices, it exceeds 80% of the maximum possible throughput of 3200MBPS per SSD, which is more than 20GBPS. This is beyond what can be expected even with a 100GbE link.

[0129] Next, the conventional access throughput of the checksum is shown in Figure 9, and the offloaded throughput of the checksum is shown in Figure 10 (the dotted lines indicate the theoretical maximum throughput of the network and SSD). For example, in Figure 9, conventional access is evaluated using pure software CRC-32C, ISA-L accelerated CRC, and AVX accelerated highway hash, which are all representative hashes used for bit rot detection. Naturally, as shown in Figure 9, conventional access is almost immediately bottlenecked by the 40GbE link. However, as shown in Figure 10, the offloaded checksum scales with the number of devices being read, bypassing not only the 40GbE link, but also the hypothetical 100GbE link. The non-accelerated CRC ends up lagging a bit, but that is an implementation issue. The highway hash and ISA-L CRC are calculated at speeds close to those of the SSD. As a result, these bit rot checks can be performed more efficiently and frequently, not only avoiding the consumption of limited available network resources, but also because they are actually performed at high speed, reducing the risk of data loss.

[0130] Additionally, although not shown in this diagram, offloading checksums reduces network traffic by over 99% due to the simple fact that instead of moving large amounts of data from the target to the host, we simply return the digested results.

[0131] To demonstrate the solution end-to-end, checksum offloading can also be integrated into object storage solutions such as Ceph, Minio, and the Swift stack. In addition to the traffic reduction mentioned above (>99%), we achieved a 2x improvement in bitrot check throughput. Furthermore, integrating this computational storage approach into these solutions is relatively easy. For example, using the tools and libraries mentioned above, the integration can often be achieved with less than 20-30 lines of code, and some of the complexity of the bitrot check function can also be removed, potentially reducing its size.

[0132] While the performance evaluation above focuses on bit rot detection as a use case, there is an infinite number of computational functions that can be offloaded to storage using this solution. These range from application-specific use cases such as log-structured merge tree (LSM-tree) compaction for key-value stores and image resizing for classifiers (e.g., the Caffe suite reached a 90% reduction in storage traffic) to more general tools such as word counting, sorting, and compression. The performance gains achieved by this solution are significant, and the modular architecture makes it easy to add new use cases.

[0133] The aforementioned solution also addresses various challenges associated with computational storage, such as handling striped data, such as data with erasure codes. As we will see further below, striped data can be handled by recognizing that many types of data sets (e.g., CSV) have an inherent structure. The offload module can therefore easily recognize this structure and distinguish when it has a complete record it can work with and when it needs to coordinate with the host or other targets to reconstruct the record.

[0134] Also challenging are handling large results and "chaining" operations where the output of one operation is immediately input to another. As we'll see further below, these challenges can be handled using a pre-allocated pool of storage, which simplifies host and target coherency issues like page cache coherency, and amortizes the cost of allocating space for results over many computations.

[0135] Furthermore, further performance improvements can be achieved by leveraging scheduling and work placement logic. For example, not all computations are suitable for compute storage, and applications may be able to work with compute storage to selectively offload some computations to maximize throughput, maximizing throughput while minimizing resource contention.

[0136] The aforementioned solutions can also improve the various security aspects of computational storage. While modern transfer protocols provide a basic level of security, additional protections such as encryption can be used to ensure that computational storage does not leak information. Another interesting advantage of computational storage is that it can reduce the amount of data "in flight", thereby reducing the risk of data leakage.

[0137] With the I / O gap between storage and network widening, as well as the increasing capacity of storage devices themselves, the only way to truly leverage modern storage performance is to put computation closer to the data. The aforementioned solution leverages the existing momentum of block-based storage, uses virtual objects that do not interfere with traditional block storage applications, and can be easily integrated into applications using simple libraries. It can reduce I / O traffic by up to 99% for various computations while transparently coexisting with existing legacy applications.

[0138] Additional embodiments relating to computational storage are described in the following sections.

[0139] [Parallel and simultaneous storage offload of replicated data in distributed storage systems] Data replication is commonly used in scaled-out, software-defined distributed storage systems, such as open source solutions like Red Hat's Ceph and Open Stack's Swift, and private solutions like Alibaba's Pangu. Storing data reliably is a necessity for all cloud service providers (CSPs). CSPs may allow users to choose the number of replicas at the expense of extra capacity. Although replication increases capacity, it is often preferred due to improved read latency and easier recovery without reassembly and reconstruction costs compared to erasure coded approaches. On the other hand, given the industry trend to move towards distributed storage for better resource modularity and independent scalability, it is attractive to adopt compute-in-storage or near-data computing techniques as described throughout this disclosure.

[0140] Figure 11 is a simple illustration of a typical distributed storage with data stored using 3-way replication. In the illustrated example, computation offload is always directed to a full copy of the target object file "foo" residing on one of the replica nodes (e.g., the left-most node).

[0141] To understand Figure 11, the following terms are defined and further referenced in this section: Client: This is where the application that utilizes the storage service to get / put files from / to the storage is running, such as an S3 API-based application. For storage that can offload computation, it is also assumed that the client can issue commands to perform the offloaded tasks ("grep" in this example). Host: This is where the interface of the storage service is presented to the client. The client's get / put / compute requests are translated into the corresponding storage implementation specific protocol related requests. In Ceph case, the "host" is usually a RADOS Gateway, a VMM host with RBD (e.g. QEMU), or a Ceph FS driver. In Swift case, this is usually a Swift Proxy server. SDS: This is a Software Defined Storage (SDS) Node. This is usually the core entity in a distributed storage system, responsible for actually placing / retrieving the client data based on the required policies such as replication and erasure coding. In Ceph, this is called an OSD Node, and in Swift, this is called a Storage Node. Target: Generally refers to an NVMeOF target in an NVMeOF fabric, or a SCSI target in a SCSI over Fibre Channel fabric. In a distributed storage environment, this is the entity where the actual storage devices reside.

[0142] The Compute-in-Storage architecture moves selected host computations to storage nodes. Examples include high-level functions (such as Hadoop's map-reduce), simple FS operations (such as search), and low-level storage system operations (such as CRC-32C integrity check). In FIG. 11, as an example, assume that a given string "bar" is to be searched for in a target object file "foo" using, for example, the Linux (registered trademark) "grep" command. "foo" exists in a 3-way replicated distributed storage. When using Compute-in-Storage, computation offloading requires constructing a computation descriptor for the computation. In general, the computation offload descriptor is as follows:<op, key, object> In this example, desc =<op: grep, key: bar, obj: foo> It becomes.

[0143] Since "foo" has three full replicas, this operation can use any replica storage node. However, offloading in these architectures typically requires one replica holder (e.g. the primary) to do the work while other replica storage nodes are idle, which is an opportunity cost in achieving parallel concurrency.

[0144] Without loss of generality, Figure 12 shows a simple example of a 3-way replicated file “foo” with a total size of 3KB. Here, the previous lookup can be performed in parallel across all three data replicas of “foo”, ensuring that compute-in-storage nodes or storage nodes with computations close to the data are never idle, and also resulting in faster response times since each replica node processes a much smaller problem space.

[0145] The present invention utilizes the existing native data parallelism through data replication to realize parallel concurrent computation offloading in distributed storage systems with data replication, which is built on top of any conventional generic computation offloading mechanism and is also transparent, allowing flexibility in optimal parallelism adapted to actual input parameters to computation offloading tasks.

[0146] There are several existing computation offload solutions (e.g., Eideticom, NGD Systems, etc.) built on FPGA or on-disk solutions for computation offload. All of these solutions target data within a single disk. Amazon's S3 Select storage feature seems to have an offload function that allows filtering of data on the S3 side instead of handing it over to the user. Existing map-reduce frameworks such as Hadoop and Spark are designed to exploit parallelism in computation tasks, regardless of whether there is data parallelism in the storage backend. However, none of these existing solutions can perform concurrent parallel computation offload on replicated-based distributed storage. For example, many of the existing offload solutions (e.g., Eideticom and NGD Systems, etc.) only focus on offloading to a single disk. Amazon's S3 Select cannot provide offloading that considers data parallelism at the storage system level. Existing map-reduce frameworks do not consider computation offloading in their entire architecture, so they cannot exploit data parallelism in the storage backend.

[0147] The present invention utilizes the existing native data parallelism by data replication to realize parallel simultaneous computation offload in a distributed storage system using data replication. It is built on the existing generic computation offload mechanism and is transparent. It extends the existing computation offload descriptor and introduces a parallel computation offload descriptor. It also defines an algorithm including range adjustment function and parallelization factor function to realize optimal parallel offload according to the target input data. It is also safe even if the storage system is in degraded mode (one or more replica nodes are not working).

[0148] This solution offers several unique advantages: (1) Realization of compute-in storage for distributed storage. It is strategically important to establish leadership in NVMeOF-based distributed architecture using data center platform architecture and non-volatile storage products. In particular, this solution realizes compute-in storage based on the large-scale use of platforms and storage products that support NVMeOF-based distributed storage. More precisely, in order to adopt high-density, large-capacity SSDs, it is very important to be able to perform compute-in storage in parallel. (2) Scale-out of Compute-in-Storage. An aspect of the features of the present invention is that it can utilize the existing native data parallelism due to data replication that exists in most distributed storage systems. This feature allows Compute-in-Storage to scale with the distributed storage system itself. (3) Robust to degraded storage. Another feature of the present disclosure is that it aims to dynamically rely on adaptive parallelization factors, making it resistant to degraded modes of the storage system where some replica nodes go offline. This allows the computational offload capabilities to be fully utilized in parallel as much as possible, even when the storage system is operating in a degraded mode. (4) Improved TCO (total cost of ownership). Furthermore, the present invention enables improved resource utilization efficiency in compute-in-storage-enabled distributed storage systems where data replication is commonly adopted. In other words, the present invention amortizes the unavoidable costs of data replication. Such TCO reduction is highly welcomed by entities such as cloud service providers (CSPs) and carriers, as it allows them to reduce data center costs and, in turn, reduce costs to their customers.

[0149] In some embodiments, a first precondition (Precondition 1) regarding input length restriction is used. For example, as a precondition, the compute-in storage offload function is activated only if the following condition is met:

number

[0150] Additionally, in some embodiments, a second precondition (Precondition 2) on active replica nodes is also used. This precondition is on the replication factor. For example, assume R is the set of replica SDS nodes for a given input object. Then, the common replication factor of that object is given by r=|R|. For example, in FIG. 11, R(object='foo') = {SDS1, SDS2, SDS3}, r=3. For the Red Hat Ceph example, the sample output for object file "foo" is shown below: > ceph osd map test foo osdmap e260 pool 'test' (4) object 'foo' -> up [2,1,3], acting [2,1,3]

[0151] In the diagram above, "foo" has three replica SDS nodes "1, 2, 3", with "2" being the primary Ceph OSD SDS node. a Let R be the active replica set that represents the set of nodes for a given input object. Clearly, when the storage system is operating normally, R = R a If the storage system is running in degraded mode, some nodes may fail and must be taken offline for recovery. a changes depending on the actual input object. The host in Fig. 11 is aware of the health of the running distributed storage system, so the replica node R a Information about the current active set of active replica elements R is always readily available to the host. a | is used as the input replication factor r of the parallel computation offloading mechanism described in the following section, as shown in the following equation:

number

[0152] This allows us to build a solution that is independent of the number of available replica nodes, making parallel computation offload more robust even when the storage system goes into degraded mode.

[0153] The basic idea of ​​this solution is a divide-and-conquer approach. In particular, this solution leverages existing data parallelism through a divide-and-conquer approach on the target file.

[0154] Figure 12 illustrates how divide and conquer works for an example of a 3KB file "foo" with 3 replicas. The computation work for the target file "foo" can be split evenly based on the replication factor. In this case, the replication factor is 3, and each storage replica node will have computation tasks for 1 / 3 of the target file "foo".

[0155] For example, assume that the size of the target file “foo” is 3KB, as shown in Figure 12. Since all replica nodes have the same copy of file “foo”, our solution effectively exploits the inherent data parallelism by tripling it by distributing the entire file “foo” evenly among the three replica nodes. In other words, we split the target file “foo” into three chunks, and the three replica nodes can perform the same computational task on 1 / 3 of the entire file “foo” in parallel. This effectively reduces the total computation time by at least r times, where r is the active replication factor explained earlier.

[0156] As mentioned above, a compute offload descriptor can be generalized as the following tuple: Compute Offload Descriptor or

number

number

[0157] It should be noted that after the target file “foo” is split into r chunks (in Fig. 12, r=3) among r replica nodes, data will be spread across adjacent chunk boundaries between the computation tasks of replica 1 node and replica 2 node (e.g., [1KB - d, 1KB + d] (d > 0)). More precisely, the actual ranges assigned to replica nodes should take into account the possibility that input parameters may appear across adjacent chunk boundaries. Fig. 13 shows that the input parameter “bar” appears on the boundary of adjacent chunks assigned to replica 1 node and replica 2 node. For example, in Fig. 13, the string “bar” occupies byte offsets 1024, 1025, and 1026 (1-based addresses).

[0158] Therefore, this solution also adds a mechanism to adjust the range based on the length of the input parameters. The range adjustment is defined based on the following range adjustment (RA) function:

number

number

[0159] Depending on the nature of the computation task, the function RA() is defined differently and can be set at run-time. However, in most cases, the target input data length will be general enough to cover most common scenarios. It should also be noted that the RA function can be done on the host side or on the target side. When done on the host side, the host translates the client's computation offload request to initiate a parallel computation offload request to the storage node, which creates a computation offload descriptor accordingly. When done on the storage node side, the storage node obtains the computation to do RA() and applies it on the fly when constructing the parallel computation offload descriptor.

[0160] Let the replication factor of an object file be r, and we want to maximize the existing data parallelism. However, the replication factor r depends on the size of the input parameter "key". We define the parallelism factor p as follows:

number

[0161] In Equation 7, "￥" is the integer division operator. Equation 7 has the following implications. First, when p = r, if the input parameter "key" is relatively small compared to each of the r chunks of the target object file, full parallelism can be achieved from all available r replicas. In other words, the "range" parameter of the parallel computation offload descriptor must not exceed the length data of each chunked replica node. Next, when exceeding the length data, i.e., when Len(key) > Len(object) / r, it falls into partial parallelization with p < r. Finally, when the input parameter "key" is very large, i.e., when Len(key) > Len(object), c = 0, meaning there is no available parallelism, and thus it returns to the original offload case as depicted in Figure 11. Here, to compute a full copy of the input object, i.e., "foo", an offload request is sent to a single target node.

[0162] Since the parallel factor p <= r, the exact number of replica nodes involved in this parallel computation offload should also be equal to p. R p Let R be the set of parallel computation replicas for a given input object. Then R p is a subset of R(object), and it is obvious that |R p | = p. In many cases, the full replication factor r can be used as p. When p < r, the host needs to perform an operation of selecting a subset of SDS nodes of size p from the existing set of all replica SDS nodes of size r. The selection can simply be the first p of r. Also, the host can increment a counter for the number of ongoing computation offload tasks at each SDS node and select the p least occupied among the r.

[0163] R p After R is generated, R pEach SDS node in is given an index i=1~p, and the i-th SDS node is responsible for the i-th range of Len(object) / p. For example, as shown in Figure 11, Rp[1]=SDS1, range[1]=[1,1KB], Rp[2]=SDS2, range[2]=[1KB,2KB], Rp[3]=SDS3, range[3]=[2KB,3KB]. The host is R p It should be noted that the SDS nodes in the,single-hop node,then send the range adjustment function output,d,to the SDS nodes in the,single-hop node,then,send the range adjustment function output,d,to the SDS nodes in the,single-hop node,then,apply the range adjustment,d,on top of the received range to correctly construct the,parallel compute offload descriptor for the target accordingly.

[0164] In the existing non-parallel computation offloading case, the response is returned from the target to the SDS node (storage node) and then sent to the host server. In the present invention, all SDS nodes involved in parallel computation respond to the host responsible for first initiating the computation offload request to these SDS nodes. The response includes the result of the computation task and where in the object file it was obtained. This is so that if there are multiple results in the overlapping boundary, in other words d in the number 2, these results can be correctly aggregated by comparing the addresses (where the multiple results are).

[0165] Fig. 14 shows an end-to-end parallel computation offload architecture. In particular, Fig. 14 shows an architectural view of the computation flow from a client application to a distributed target. Fig. 15 shows a flow chart of the parallel computation offload computation in the host process and the SDS node process.

[0166] Note that in some embodiments, requests from hosts to storage SDS nodes may be consolidated by multicasting the requests, which can further increase parallelism. To multicast such requests, the range parameter needs to be modified as a range list, from which each SDS node appropriately selects its own range.

[0167] 16 shows a flowchart 1600 for performing computational tasks on data stored using replicated data storage. In various embodiments, the flowchart 1600 may be implemented using the embodiments and functionality described throughout this disclosure.

[0168] The flowchart begins at block 1602 with receiving a request to perform a computational task on a data object stored in a storage system, the data object being replicated on each storage device in the storage system.

[0169] The flowchart then proceeds to block 1604, where it identifies multiple chunks of the data object to be processed in parallel on the storage system, such that a computational task is to be performed on the multiple chunks by multiple storage devices.

[0170] The flowchart then proceeds to block 1606, where it sends a plurality of computational offload commands to a plurality of storage devices, each computational offload command instructing a corresponding storage device to perform a computational task on a corresponding chunk of the plurality of chunks.

[0171] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may resume at block 1602 to continue receiving and processing requests to perform computational tasks on data objects stored in the replicated storage system.

[0172] [Offloading computation to distributed striped storage] In-storage compute is an attractive way to eliminate I / O bottlenecks because it allows operations to be performed "close" to the data, instead of pulling it across constrained I / O links (e.g., network). A host server connected to a target storage server using NVMeOF (NVMe over fabrics) is one example where moving compute from the host to the target is desirable.

[0173] However, current compute-in-storage designs incur high network costs because they assume that all data is fully discretized into easy-to-interpret chunks or replicas, fully located on the same logical device (e.g., Hadoop+HDFS), or that there is a central administrator collating data from multiple locations. For example, to offload "search" operations to the target storage server, existing solutions assume that the storage server has access to the entire file or object, so that it can perform the search close to the storage and only return matching results to the host. Such solutions obviously do not work for example with erasure-coded storage, where the storage system (e.g., server or individual device) only sees a portion of the data.

[0174] While many storage systems stripe data across multiple locations for performance and reliability reasons, new mechanisms must be devised to offload computation to individual fragments (portions of a data object), especially in systems using block storage. For example, in a striped storage system, a data object may be divided into stripes, or smaller units of data, and distributed across a collection of storage devices in the striped storage system. Thus, a "fragment" of a data object may refer to a contiguous portion of the data object that is stored contiguously or consecutively on a particular storage device in the striped storage system.

[0175] This solution allows systems with data distributed across multiple locations to obtain many of the compute-in-storage benefits of purely replicated or single-copy storage, while minimizing the coordination required between separate storage locations. It leverages the inherent structure of most data and returns only partial residual data elements that may span storage targets along with the desired result. In other words, you can stripe a large file across n storage targets, with each storage target seeing 1 / n of the entire file, yet still offload computation to each piece separately and collect the results and residuals.

[0176] For example, consider the word count use case: each target server counts the words in its own fragment and returns the result to the host. The host can then add up the word counts across fragments to get a total value for the file. Words that cross storage servers become "residuals" that are returned to the calling host to be reconstructed into complete words for the final count. These residuals are simply strings of characters that start and end on non-space boundaries, so determining whether they are the end of a word is trivial and easily identifiable.

[0177] Figure 17 shows an example of a word count residual: the word "counting" straddles a boundary, but because the left chunk sees "coun" with no space after it, it can be identified as a residual independently, just as the right chunk sees "ting" with no space before it.

[0178] Furthermore, the advantage of this distributed computing is that it can take advantage of the distributed nature of data storage to operate on data in parallel, resulting in faster performance.

[0179] The solution recognizes that many data sets, while logically stored as a single entity (e.g. a CSV file), are composed of many smaller, well-structured elements, like individual rows or records in a CSV file. It leverages this structure to perform computations on top of the distributed data. Individual data elements are computed, and any data elements that cross the target server boundary are returned to the client where they can be reassembled and queried. This allows the solution to achieve the full benefits of compute-in-storage (reduced data movement) without having to fully centralize the data.

[0180] This opens the door to compute-in-storage applications that can leverage a variety of hardware, from commodity CPUs and FPGAs to specialized hardware and accelerators, such as neural network hardware. In particular, the solution can utilize compute-in-storage to work with erasure coded data.

[0181] To begin with, we define some vocabulary to describe distributed storage in general: A data object (e.g. a file) consists of multiple chunks. These chunks are typically distributed across multiple machines for reliability and performance. These chunks may be erasure coded (e.g., distributed with parity), but this is not required. Note that because this solution does not depend on the presence of parity data, parity data is not included in the example diagrams of this solution. · Organize groups of chunks into stripes. This logical arrangement allows the parity of the erasure codes to be easily calculated and read in parallel. A block is a linear grouping of chunks of data arranged across stripes. If the data does not fill the block, it is padded. The size of the blocks can vary, but as long as the size is known it does not affect the solution. It should be noted that these "blocks" are different from the blocks seen on block-based devices such as SSDs. Unless otherwise noted, in the following description, the term "block" refers to the higher level of abstraction described here. The part of a block that exists inside a chunk is a subblock. A boundary is where a piece of data transitions to a chunk in another place.

[0182] A simple example is shown in Figure 18. Figure 18 shows an example data layout consisting of 2 stripes with 4 chunks per stripe. Each stripe has 2 blocks, and each block has 4 sub-blocks. Since we are only interested in plaintext data here, no underlying data blocks or chunks are shown.

[0183] The prototype and layout work is based on the Minio system, but uses a more general vocabulary. The principles and conditions apply without loss of generality to other storage systems, such as Ceph and Swift.

[0184] The solution also leverages block-based compute-in-storage systems (e.g., object-aware storage and virtual objects) to allow arbitrary computation to be performed on block-based storage.

[0185] The essence of this solution is to examine each chunk and its constituent sub-blocks independently and return the required data. Any remaining data elements across the boundary are returned to the caller (e.g. the host server) to be combined and examined, allowing the entire object to be traversed while reducing data movement.

[0186] To highlight how this solution works, consider the example in Figure 19, which has the same layout as Figure 18, but with a block size of 12 characters and a subblock size of 3. For example, the object shown in Figure 19 contains the sentence "The quick brown fox jumped over the lazy dog." In this example, the second block of chunk-2d contains padding, and the word "fox" straddles a chunk boundary (e.g., the boundary between the second blocks of chunk-1b and 1c).

[0187] The layout chosen is representative of some object storage systems that write data out in blocks, in a word-like pattern across pages, often to avoid the system having to buffer large amounts of data before persisting it to storage.

[0188] In a traditional system, searching for the word "fox" would require centralizing all data and scanning the reconstructed strings to find the specific word. A naively distributed search would miss the search string because "fox" straddles the boundary between chunks 1b and 1c. However, the solution above tells the Compute-in-Storage system where the boundaries are and the maximum possible size of a data element. In this example, the word "fox" is a string of length 3.

[0189] In this example, most of the chunks and their sub-blocks do not match any characters in the search string and can therefore easily be discarded independently without any adjustments. However, chunk 1b's sub-block matches the first two characters that should be returned to the host and / or client, and similarly chunk 1c matches its first character x. Neither can be 100% sure if they found the search string, so they return the data to the calling application on the host to check. These two chunks return characters from their respective sub-blocks, and the calling application (running on the host server) does the final comparison, as shown in Figure 20. For example, Figure 20 shows an example of returning ambiguous residual data for the final reconstruction.

[0190] Central to the solution is the identification of so-called residuals: data elements that cross chunk boundaries (e.g. CSV records) and that need to be combined with other residuals to be meaningful. Tail residuals are sub-elements that cross boundaries at the end of a sub-block, and head residuals are sub-elements that are at the beginning of a sub-block. These residuals may be returned to the calling client in addition to any other data of interest, where they can be reconstructed into complete data elements and operated on as normal.

[0191] There are two basic ways to identify the residuals. (1) Fixed Size: If the data elements are all a fixed size of N bytes, then it is trivial to simply return the first and last N-1 bytes of the subblock on either side of a data boundary, even if there is no underlying structure to exploit. (2) Structured / Semi-structured Data: Some data is structured, while other data has clear separators between consecutive elements, for example, carriage returns in CSV and curly braces in JSON. (i) Tail residual: if an element is truncated at a sub-block boundary, return all data from the end of the previous structure (e.g., a line in CSV) to the end of the sub-block. (ii) Leading Residual: If an element is "too short" (a fragment was missed), identify the beginning of the next complete element. Return all data from the beginning of the subblock to the beginning of the first complete element.

[0192] Additionally, applications must have the ability to accurately describe their data layout to the compute-in-storage system, for example by providing a set of offsets that represent the boundaries and which boundaries map to each other. For example, in Figure 19, it should be recognized that subblocks 1c and 1d, 1d and 2a, etc. are logically adjacent. This information is commonly available as metadata in many systems (e.g., xl.Json files stored alongside objects in Minio).

[0193] Figure 21 shows a simple example of identifying residuals in semi-structured data, where the highlighted CSV value crosses a boundary and is not terminated with a carriage return (\n), making it recognizable as an incomplete record.

[0194] In terms of request flow, the host (e.g., the calling application) uses the layout metadata to build a list of boundary entries that are used to organize and sort the residuals. Each boundary entry contains a reference to a file or object chunk, offsets indicating the start and end of a subblock, a buffer to store the tail or head residual, and a pointer to the logical contiguous region. Example residuals and an example boundary list are shown in Figure 22 and Figure 23, respectively.

[0195] For example, Fig. 22 is a diagram showing an example of residuals when searching for a character string of length 2, and Fig. 23 is a diagram showing an example of a boundary list having a residual return for stripe 1 in Fig. 22. In this example, since a character string of length 2 is being searched for, if there are residuals at the beginning and end, they will each be of length 1 (e.g., n-1).

[0196] During this construction, we also check all the boundaries that occur for each chunk, so that at each chunk location we can compute and retrieve all the residuals in one request.

[0197] These grouping boundaries are contained in the computation descriptor, which for example contains the list of blocks that make up the file object, as well as the operation to perform (e.g., search) and arguments (e.g., "fox"). Additionally, the descriptor is extended to include the boundaries and offsets of each chunk.

[0198] Each machine on which a chunk is placed performs the operation independently and returns a residual specific to the requested operation. As mentioned above, the complete data is returned as usual along with these partial residuals. These residuals are reconstructed and operated on the host side before returning the results to the calling application.

[0199] The overall flow can be summarized as follows: (1) Host: The host obtains layout metadata (e.g., Minio’s XL.json) from the storage target. (2) From host:layout metadata (i) Calculate the bounds and offsets and create an initial bounds list. (ii) At the same time, match the offsets of each chunk. (3) Host: Create a compute descriptor and submit it to the target. (i) It should be noted that each chunk has its own descriptor created. (4) Target: Each target processes each descriptor. (i) Residual data is calculated and arranged for return as described above. (ii) Non-residual data is operated on and the relevant results are arranged and returned. For example, in the case of CSV, complete CSV rows can be queried as usual and matching values ​​returned. (5) Host: The residuals are returned, reassembled and computed (e.g., as shown in Figure 19). (6) Host: The result is returned to the calling application.

[0200] 24 illustrates a flowchart 2400 for performing computational tasks on data objects stored in a striped storage system. In various embodiments, flowchart 2400 may be implemented using the embodiments and functionality described throughout this disclosure.

[0201] The flowchart begins at block 2402, which illustrates receiving a request to perform a computational task on a data object stored in a storage system, the data object being striped across multiple storage devices in the storage system.

[0202] The flowchart then proceeds to block 2404, where it identifies a plurality of fragments of the data object on a plurality of storage devices, each fragment consisting of a contiguous portion of the data object stored on one of the plurality of storage devices.

[0203] The flowchart then proceeds to block 2406, where it sends multiple computational offload commands to multiple storage devices, each computational offload command instructing a corresponding storage device to perform a computational task on a corresponding fragment stored in the corresponding storage device.

[0204] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may restart at block 2402 to continue receiving and processing requests to perform computational tasks on data objects stored in the striped storage system.

[0205] [Compute-in-storage on a distributed storage system operating in degraded mode] In the context of rack-scale storage design (e.g., as shown in Figure 26), compute-in-storage solutions move data-intensive computations (e.g., filtering, searching, compression) to storage targets running protocols such as Internet SCSI (iSCSI) or NVM over Fabrics (NVMeOF). The goal is to reduce the network bandwidth required to perform these tasks on the storage host server. In the case of erasure coding (EC) storage, which is gaining importance due to capacity efficiency at scale, data is erasure coded across storage nodes. However, if one or more storage nodes fail, the storage system goes into degraded mode and data must first be reconstructed on the storage node side and then pulled from the target side. As a result, client applications cannot fully benefit from compute-in storage (e.g., when copying data over the network for reconstruction). This problem can be solved by introducing a mechanism that temporarily assigns a storage node as a degraded-mode compute offload storage node, which creates a degraded-mode compute descriptor and enables the target to reconstruct data from parity and perform the requested compute-in storage task.

[0206] Most of the existing industry distributed cloud storage systems support EC storage, such as Minio's private cloud storage, Red Hat Ceph, Open Stack Swift, and Aliyun Pangu. EC is commonly expressed in terms of EC(k,m), where the input data is divided into k data chunks and m parity chunks are calculated based on the distribution matrix. Even if less than m chunks are lost (e.g., m erasures) due to system crashes or server failures, the original data can still be reconstructed from the surviving k chunks. For example, Figure 25 shows an example of the logic flow for erasure coding construction and reconstruction. In Figure 25, the logic flow for EC construction and reconstruction is shown in sub-figures (a) and (b), respectively, and a (3,2) scheme with three data chunks and two parity chunks is taken as an example of an EC scheme.

[0207] Meanwhile, the industry is moving towards distributed storage to achieve better resource modularization and independent scalability, especially with the growing storage capacity needs from a huge number of clients at cloud scale. In distributed storage, storage server nodes (e.g. Ceph OSD nodes, Swift Storage nodes, or Minio Server nodes) are initiators (e.g. NVMeOF or iSCSI) with storage located at remote targets. Naturally, this approach consumes a lot of extra network bandwidth and increases latency as data travels back and forth between the target and the storage server side.

[0208] Therefore, in the following description, a mechanism for realizing compute-in-storage on an EC-based distributed storage that operates in a degraded mode in which data and parity are erased is described.

[0209] Figure 26 shows an example of compute-in-storage in a distributed software-defined storage system with EC(3,2) deployed on NVMeOF distributed storage. It should be noted that the offloaded computation is always assigned to all plain data chunks (y1, y2, y3) of "foo". The two parity chunks (y4, y5) are not involved in this process.

[0210] To understand Figure 26, the following terms are defined and further referenced in this section: Client: An application that uses the storage service (e.g., through S3, block, filesystem interfaces, etc.). For storage that allows compute offloading, it is also assumed that the client can issue commands to perform the offloaded tasks ("grep" in this example). Host: Here, the interface of the storage service is provided to the client. The client's get, put, and / or compute requests are translated into the corresponding storage implementation specific protocol related requests. For example, in Ceph a "host" is called a RADOS Gateway or QEMU RBD, in Swift a "host" is called a Swift Proxy Server, in Minio a "host" is called a Minio Server. SDS: A Software Defined Storage (SDS) Server Node. This is usually the core entity in a distributed storage system, responsible for actually placing and retrieving client data based on the required policies such as replication and erasure coding. In Ceph this is called an OSD node, in Swift this is called a storage node. In Minio this is simply called a Minio Server Node. Target: Generally refers to an NVMeOF target in an NVMeOF fabric, or a SCSI target in a SCSI over Fibre Channel fabric. In a distributed storage environment, this is the entity where the actual storage devices reside.

[0211] The Compute-in-Storage architecture moves selected host computations to storage nodes. Examples include high-level functions (such as Hadoop's map-reduce), simple FS operations (such as search), and low-level storage system operations (such as CRC-32C integrity check). In FIG. 26, as an example, assume that a given string "bar" is to be searched for in a target object file "foo" using, for example, the Linux "grep" command. "foo" resides in an EC(3,2) erasure coded distributed storage. When all plain data chunks (i.e., y1, y2, y3) are available, the computation offload will build a computation descriptor for the computation on each plain data chunk, which is generally represented as the following: Compute Offload Descriptor:=<op, key, object> In the example in Figure 26, desc =<op: grep, key: bar, obj: chunk for foo> It becomes.

[0212] However, when the system is in degraded mode due to disk failure or server crash, the erasure coding scheme EC(k,m) allows up to m erasures out of a total of k+m chunks, but there is no solution to realize compute-in storage when the m erasures involve any of the k data chunks. In the case of Figure 26, if any of the erased chunks come from d1, d2, or d3, compute-in storage cannot be performed until the lost chunks are reconstructed. That is, in degraded mode, the availability of the compute-in storage service is lost, even though the availability of the storage service is guaranteed.

[0213] Existing solutions do not allow for compute-in-storage in degraded mode for EC-based distributed storage. For example, when a storage system goes into degraded mode, it is possible to assign the computation task to one of the still-operating SDS nodes (e.g., using an approach like Storelet). However, as can be seen from Figure 28, if we try to reconstruct data using SDS4 when y2 is lost, we first need to hand over the surviving chunks of y1, y3, and y5 from the target side to SDS1, SDS3, and SDS5. Then, SDS4 needs to pull y1, y3, and y5 from SDS1, SDS3, and SDS5 to reconstruct the lost y2. It is easy to imagine the bandwidth waste caused by data movement from the target side. At the same time, while adding computation to SDS4, the existing compute-in-storage capabilities on the target side are completely idle.

[0214] The solution presented below realizes compute-in-storage on distributed storage using an erasure coding scheme EC(k,m) for any m erasures. The mechanism leverages direct target-to-target communication on the target side and has the following main components: (1) a process to select SDS nodes as degraded-mode compute offload SDS nodes, (2) a mechanism to create degraded-mode compute offload descriptors.

[0215] This solution offers several unique advantages: (1) Realization of compute-in storage for distributed storage. It is strategically important to establish leadership in NVMeOF-based distributed architecture using data center platform architecture and non-volatile storage products. In particular, this solution realizes compute-in storage based on the large-scale use of platforms and storage products that support NVMeOF-based distributed storage. (2) Improved availability of compute-in storage services. The solution enables compute-in storage services even when the storage system is in degraded mode. This is especially important for cloud customers with large storage deployments that regularly experience disk failures and server crashes. This allows compute-in storage services to be provided to customers even when the storage is in degraded mode, reducing the possibility of SLA violations for cloud service providers (CSPs). (3) Improved TCO (total cost of ownership). Moreover, the solution allows for more efficient resource usage in a distributed storage system that uses erasure codes and is enabled by compute-in-storage, since the compute-in-storage can continue to serve its customers while the storage system is being restored. In other words, the solution reduces the TCO of entities such as CSPs and Telcos, since the inevitable costs of data restoration can be amortized over erasure chunks, lowering the data center costs and, therefore, the customer costs.

[0216] In an EC(k,m) erasure coding scheme with k plain data chunks and m parity chunks, if a lost or erasure chunk is contained in any of the m parity chunks, then all k data chunks are still available in the stripe, so this problem has already been addressed by other solutions in this disclosure.

[0217] The solutions below address EC storage for erasures from the plain k data chunks, or any m erasures from all the encoded k+m chunks, both of which require reconstruction via matrix inversion. Thus, in the following discussion, when we refer to e erasures (e <= m), the n erasure chunks are either lost plain data chunks or lost encoded chunks (i.e., encoded data and parity).

[0218] SDS nodes are generally responsible for constructing a compute offload descriptor and sending it to the corresponding target, as shown in FIG. 26. In the degraded mode, not all SDS nodes are available for this collaborative compute offload work. For example, in FIG. 27, SDS2 is unavailable because target2 has crashed. Apart from the normal operation SDS nodes that still hold the data chunks (e.g., SDS1 for y1 via target1, SDS3 for y3 via target3), called normal mode compute offload SDS nodes (NMCO SDS), the host needs to select an SDS node, called degraded mode compute offload SDS node (DMCO SDS), to perform the compute offload operation on behalf of SDS2. Furthermore, the selected degraded mode compute offload SDS node is responsible for constructing a compute offload descriptor (called degraded mode compute offload descriptor) that is different from the normal compute offload descriptor, which will be described later.

[0219] In the following, an embodiment of the selection algorithm based on tracking the computation offload load index at each SDS node of the host is described. First, L is used as the computation offload load index of each target i instructed by the corresponding SDSi node to perform the computation offload task. i For each computational offload task, we simply use L i is defined as follows:

number

number

number

[0220] The host tracks the load of each SDS node i and when a new task j is started, it calculates the cost l ij L i and when the task is completed, the cost ij L i With the above definition, one SDS node can contribute as an NMCO node for one task and as a DMCO node for another task, allowing the computation offloading tasks to be balanced among all SDS nodes.

[0221] Once the above load indicators have been established, the selection process can be described as the following steps: (1) The host finds the set of SDS nodes that own all data chunks and calls them S k Let us assume that. (2) The host finds the set of SDS nodes that own all the parity chunks and calls it the S m Let us assume that. (3) The host is S k The set of SDS nodes for erasure in, in other words, S k Find the node where the service is stopped and set it to S e Let us assume that. (4) The host is S k ￥S e Designate all nodes in as normal mode compute offload SDS nodes (NMCO). (5) The host creates a set S based on the current offload load indicator L. m Sort. (6) The host is S m Some nodes from |S e Select | as the degraded mode computation offload SDS node (DMCO) with the lowest load L.

[0222] In addition, the above DMCO node selection algorithm is scalable even when the distributed storage system scales out.

[0223] As mentioned above, the compute offload descriptor for normal mode is written as follows: Compute Offload Descriptor or

number

[0224] Since the original data chunks are currently unavailable to the DMCO SDS node, it must first reconstruct them based on all peer chunks. Then, it can perform the required computation offload tasks on the reconstructed data. Therefore, the target corresponding to the DMCO SDS node must obtain the data chunks from the target of its peer SDS node. To achieve this, we define the degraded mode computation offload descriptor (dmdesc) as follows:

number

[0225] Compared to a normal descriptor that describes where a chunk is, dmdesc<peer chunk, target> Note that each peer chunk has a list of peer chunk pairs. With this information, the receiving target can perform direct target-to-target communication with its peer targets to obtain all peer chunks for EC reconstruction.

[0226] For Compute-in-Storage to function in degraded mode, the selected DMCO node must have knowledge of the targets corresponding to the peer SDS nodes. In one embodiment, this can be done via an extension of the SDS cluster configuration file, such as the OSD section of the Ceph cluster configuration file, as shown in Figure 27. For example, Figure 27 shows an example Ceph OSD configuration that extends the target information of the OSDs. The example configuration shows an NVMeOF target based on RDMA transport.

[0227] The above target information is available to all SDS nodes and can be used to construct the degraded mode compute offload descriptor.

[0228] For the communication protocol between targets, we can leverage existing approaches such as T10 XCOPY to allow targets to read from and write to other targets, provided the target information is detailed enough so that any target can find its peers.

[0229] FIG. 28 shows an example of computation offload to degraded mode storage. In particular, FIG. 28 details the flow of operations from a client application to a distributed target. In this example, the distributed storage uses an EC(3,2) scheme for data redundancy. Furthermore, the storage system is currently operating in degraded mode due to the loss of one data chunk from SDS2 / target2. In this example, if SDS2 / target2 fails, SDS4 is selected and a degraded mode computation offload descriptor is constructed. Target4 uses target-to-target direct communication to retrieve the peer chunk and reconstruct the lost data chunk "y2", and then performs computation processing on the target side.

[0230] Below, the steps are described in detail for a client that is aware of the computational offloading capabilities of the distributed storage. (1) A client sends a request to perform a computational task that is offloaded to the distributed storage, e.g., searching for the string “bar” in the file “foo”. (2) The host checks which SDS nodes are still available and which are out of service. It selects SDS1 and SDS3 as normal mode computation offload SDS nodes (NMCO). Then it checks the computation offload load indicators of SDS4 and SDS5 according to Equation (1), Equation (2), and Equation (3). Then the host decides to select SDS4 as the degraded mode computation offload node because SDS4 has a lower computation offload load. (3) The host sends a compute offload request to SDS1, SDS3, and SDS4. It flags SDS1 and SDS3 as NMCO nodes, but flags SDS4 as a DMCO node to perform degraded mode compute offload (e.g., construct a degraded mode compute offload descriptor). (4) SDS1 and SDS3 receive the request as NMCO nodes and build the computation offload descriptors for normal mode according to equation (4). SDS1 and SDS3 send the descriptors to the corresponding targets. (5) SDS4 receives the request as a DMCO node. SDS4 finds the target information of the peer SDS node, for example from the cluster configuration information. Then, SDS4 constructs a degraded mode compute offload descriptor according to equation (5). Finally, SDS4 sends the degraded mode compute offload descriptor to the target, Target4. (6) Based on the received degraded mode computation descriptor, Target4 retrieves peer chunks from its peer targets using the direct target-to-target communication protocol. For the descriptor pairs where the target is itself, it performs local I / O to retrieve the chunks. Target4 then performs EC reconstruction to recover the lost chunks. (7) Target1, Target2, and Target4 execute the corresponding computation offload tasks.

[0231] It should be noted that SDS4 could potentially request Target4 to return the reconstructed data so that this chunk of data can be placed somewhere else for permanent recovery, rather than being discarded after the compute-in-storage task is finished on the target side, which effectively makes reconstruction an offloadable operation suitable for compute-in-storage.

[0232] 29 shows a flowchart 2900 for performing computational tasks on data objects stored in an erasure coded storage system. In various embodiments, flowchart 2900 may be implemented using the embodiments and functionality described throughout this disclosure.

[0233] The flowchart begins at block 2902 with receiving a request to perform a computational task on a data object stored in a storage system, the data object having been erasure coded across multiple storage devices within the storage system.

[0234] The flowchart then proceeds to block 2904 where it is determined that the storage system is operating in a degraded mode, which indicates that the plurality of storage devices is comprised of a set of available storage devices and a set of unavailable storage devices.

[0235] The flowchart then proceeds to block 2906, where it identifies a set of available chunks of the data object on a set of available storage devices.

[0236] The flowchart then proceeds to block 2908, where it identifies a set of unavailable chunks of the data object on the set of unavailable storage devices.

[0237] The flowchart then proceeds to block 2910, which illustrates dividing the set of available storage devices into a set of normal mode storage devices and a set of degraded mode storage devices.

[0238] The flowchart then proceeds to block 2912, where it sends a first set of computational offload commands to the set of normal mode storage devices, the first set of computational offload commands instructing the set of normal mode storage devices to perform computational tasks on the set of available chunks.

[0239] The flowchart then proceeds to block 2914, where it sends a second set of computational offload commands to the set of degraded mode storage devices, the second set of computational offload commands instructing the set of degraded mode storage devices to reconstruct the set of unavailable chunks and perform computational tasks on the corresponding set of reconstructed chunks.

[0240] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may restart at block 2902 to continue receiving and processing requests to perform computational tasks on data objects stored in the erasure coded storage system.

[0241] [Chained operations with preallocated computation block storage] In this section, we describe mechanisms that enable more complex storage offloading, such as creating search indexes or Bloom filters in-situ within block storage objects, by chaining multiple compute-in-storage operations.

[0242] This technical challenge is based on certain observations related to previous block-based computational storage solutions: for example, there are currently only two ways to perform computations in-situ. (1) Multiple round trips from the host (e.g., sending a command, identifying where the result lands, sending additional commands to operate on that result, etc.). (2) Create a new compute offload that integrates the entire set into a single method. This can be problematic because it can result in duplication of development effort that could be made trivial by chaining together existing functionality.

[0243] Based on these observations, we propose below a solution that enables simple and efficient chaining of existing computational functions using pre-allocation of circular buffers to store the results of large offloads, while eliminating the need for multiple round-trips between the client and the host.

[0244] There are no prior computational storage solutions that address the problems solved by the solutions presented below. For example, while there are various solutions at the object storage level (e.g., Ceph's dynamic classes and Swift's Storelets), these object-based solutions require the use of heavyweight object interfaces and do not address the round-tripping issues inherent in result chaining. Furthermore, while the object interfaces are simple, they are inflexible and difficult to work with without requiring coordination at a higher level (e.g., the client).

[0245] The basic idea of ​​this solution is to utilize temporary buffer space to store the output and use it as input for the next operation, all in the context of a block-based compute-storage system. The solution uses a pre-allocated pool of blocks to store the results, and if the result is small enough, it simply keeps it in memory and passes it as a pointer to the next operation. In this way, compute-in-storage operations can be efficiently chained to further improve performance.

[0246] In particular, the solution implements chained operations for compute-in-storage using pre-allocated output space for virtual objects, computational descriptors, and computational storage results.

[0247] A virtual object is a transient mapping of logical blocks and lengths that describes an object. For example, the extent map of a file along with the file length is a virtual object that describes the location and size of the file.

[0248] A compute descriptor contains at least one input virtual object (e.g., the block extent map and length of an input file), plus an optional output virtual object that can store the method / function name, function arguments, and the method / function result. These compute descriptors are interpreted by the block storage object to complete a particular operation.

[0249] Additionally, the solution extends the compute descriptor to chain operations. For example, a series of operations can be chained together into an operation chain or sequence (e.g., filtering a file and compressing the result). The solution allows multiple operations to be defined and they can be processed in sequence or consecutively. The output of the first operation becomes the input of the second operation, and so on until all operations are completed. The final output may be returned directly to the host application if it is small in size, written to a specified block location (e.g., one of the virtual output objects), or written to one of the pre-allocated buffer locations mentioned above. If written to one of these pre-allocated locations, the LBA+offset+length is returned to the host, allowing the host to retrieve the data later.

[0250] Additionally, a pool of buffers is created (e.g., on block storage) to store large intermediate results. The number of pools can be equal to the number of operations that are planned to be executed in parallel. This eliminates the need for complex inter-process communication and locking between potentially simultaneously executing chains of operations. The Logical Block Addresses (LBAs) of these storage buffers are tracked by the computation engine within the storage object. A simple mutex can track which buffers are available or unavailable.

[0251] A computation descriptor for a chained operation may include the following: Virtual Objects:<extent list> <object length> ·output:<extent OR pre-allocated OR best effort direct return> Note: Even with a best effort, it is possible that the LBA location will be returned if the size is too large to pad. ·Operation chain:<operation 1> ,<operation 2> ,<operation 3> … Operation arguments:<op1 args> ,<op2 args> …

[0252] Figure 30 shows an example of the overall architecture and data flow of chained compute-in-storage operations. In the example shown, a host application issues a request to a block storage target with a compute descriptor. The compute engine reads the initial input and executes the first specified operation. If the result is greater than a configurable high water mark, it is buffered in an intermediate buffer pool; otherwise, it can be passed directly as input to the next specified operation. Once all operations are completed, the result is finally written to a result location (which can be a pre-allocated buffer location) or returned directly to the host through a forwarding payload.

[0253] Also note that even operations that are not explicitly chained can make use of intermediate buffer pools: for example, a recursive merge operation can store its intermediate results in a buffer pool.

[0254] As an example, consider the use case of in-situ Bloom filter creation, which might involve creating a Bloom filter for fast searching of tags or keywords in large text files. The input would be a virtual object describing the location and length of the text file. The output would be a "best-fit" file from a pre-allocated buffer pool. The first operation is a simple word set identifier, creating an entry for any previously unseen word. This set list is stored in an intermediate buffer, which is then used to create a new Bloom filter. This Bloom filter is then stored in the pre-allocated "best-fit" file, and its location and length are returned to the host. The host is then free to read and operate on the Bloom filter.

[0255] As another example, consider the use case of Log-Structured Merge (LSM) tree compression. LSM trees are commonly used in key-value (KV) stores (e.g., LevelDB). In these KV stores, as data is inserted or deleted, it is stored in an in-memory append-only log. As the log fills up, it is periodically sorted and dumped onto persistent storage (e.g., HDD, SSD, etc.). These sorted logs are periodically merged and compacted, for example, in a process roughly similar to a traditional merge-sort algorithm.

[0256] Typically, after data is read from the storage target into the CPU-memory complex, it must be merged and quickly written back to the target, a full round trip for the data involved, potentially over contended network links. Traditional compute-in-storage solutions can also struggle with the multiple stages required for this computation. However, the chaining solution described above can utilize a staging zone for the merge stage, simplifying the process and reducing overall memory since the compute target does not need to keep everything in memory for each stage.

[0257] 31 shows a flowchart 3100 for performing chained computational tasks on data objects stored in a storage system. In various embodiments, the flowchart 3100 may be implemented using the embodiments and functionality described throughout this disclosure.

[0258] The flowchart begins at block 3102 with receiving a request to perform multiple chained computational tasks on a data object stored in a storage system, the data object being stored on a corresponding storage device in the storage system.

[0259] The flowchart then proceeds to block 3104, where it illustrates sending a chained computation offload command to the corresponding storage device, where the chained computation offload command instructs the corresponding storage device to perform multiple chained computation tasks on the data object.

[0260] The flowchart then proceeds to block 3106, where the results of the chained computation offload command are received from the corresponding storage devices.

[0261] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may restart at block 3102 to continue receiving and processing requests to perform chained computational tasks on data objects stored in the storage system.

[0262] [Computational memory for visual data and analysis] In this section, we describe this compute-in-storage solution for visual data,analysis and other similar workloads.,For example, the proposed solution applies generally to,any workload that has input files, output files, and some form of,computation that maps input to output.

[0263] Instead of moving data from storage to the CPU (where computation typically occurs, for example), we want to move computation to the data. Until now, the benefits of this "active storage" didn't justify the cost. For example, traditional block-based storage was never fast enough to be largely I / O limited, because the limiting factor was storage access time. Now, however, the opposite is happening: storage access times are fast, but I / O paths are relatively slow.

[0264] In particular, visual data (e.g., images, photos, videos, etc.) is becoming increasingly valuable due to the latest in high-performance machine learning and data mining techniques. With this in mind, there are several points that motivate this solution to enable compute-in-storage of visual data: (1) "Striped" data, such as erasure-coded data that spans multiple targets, ignores data element boundaries. For example, a video keyframe may span target boundaries, or a list of feature vectors describing an image may contain vectors that span target boundaries. This requires complex offset calculations and data linkage, or the data must be completely centralized before calculations can be performed, negating the benefits of compute-in-storage. (2) Visual data takes up a huge amount of space. While people generally want data “close” to where it is used, this presents challenges, such as wastefully replicating / storing copies of the data on the machines that process the data, or incurring large amounts of network traffic to retrieve data that is efficiently stored in separate storage locations outside the machine. (3) Much of the visual data processing produces outputs that are significantly smaller than the inputs: for example, feature vectors describing objects in an image may be only a few kilobytes long, or the desired processed image for input to a classification model such as a neural network may be much smaller than the original image. (4) Even if compute-in-storage is possible for a given visual data, the results are often large enough that simply using a block-based compute-in-store protocol is inefficient: the results may be too large to embed in the return, requiring expensive pre-allocation of space on the storage target to write the results to.

[0265] Based on these observations, the proposed solution reduces the need for expensive data pre-allocation, provides a way to align data so that individual data elements do not exceed target boundaries, and eliminates the need for complex offset operations.

[0266] Existing solutions lack the ability to efficiently offload vision processing to block storage. In general, current solutions are overly complex and ignore the problem of expensive pre-allocation of storage for writing results, thus failing to achieve the benefits of compute-in-storage for vision data.

[0267] The proposed solution has two parts. First, we handle the pre-allocation issue by pre-allocating a large circular buffer on the storage target to write results to and / or pre-allocating a pool of files of varying sizes at system startup, so that the cost of pre-allocation is amortized and hidden from the calling application. Second, we outline how to align data in a striped system; the encoding of the data remains largely unchanged and no complex offset arithmetic or host-target alignment is required. We use padding to effectively slightly adjust the position of data elements so that individual elements do not cross the target boundaries.

[0268] The proposed solution improves distributed storage platforms (e.g., Intel Data Management Platform) and compute-in-storage for visual analytics, improving overall performance and increasing the need for compute-in-storage hardware in the existing computing ecosystem.

[0269] Below, we also outline some use cases (e.g., image processing, classification, feature vector querying and extraction) to explain how the proposed solution can be applied.

[0270] The proposed solution relies on a computation based virtual object approach with computational descriptors as described throughout this disclosure. A virtual object is simply a temporary mapping that points to the location of a particular set of data, such as a file or object. In this case, it is a block storage object or an extent list that describes the location of logical blocks that a device can interpret. This mapping is part of a computational descriptor that describes the inputs, outputs, operations, and arguments of the compute-in-storage function. The compute-in-storage object interprets these computational descriptors and acts on the instructions contained within them.

[0271] First, let’s define the following terms related to distributed and / or block-based storage: A striped object (such as a file) is made up of multiple chunks. These chunks are typically distributed across multiple machines for reliability and performance. These chunks may be erasure coded (e.g., distributed with parity), but this is not required. Note that the example solution diagram does not include parity data, as the solution does not depend on the presence of parity data. A group of chunks organized into stripes. This logical arrangement allows the parity of the erasure codes to be easily calculated and read in parallel. A block is a linear grouping of chunks across a stripe. If the data does not fit into a block, it is padded. The size of the blocks can vary, but as long as the size is known it does not affect the solution. It should be noted that these "blocks" are different from the blocks seen in block-based devices such as SSDs. Unless otherwise noted, in the following description, the term "block" refers to the higher level of abstraction described here. The part of a block that exists inside a chunk is a subblock.

[0272] First, striped environments require the simplification of compute-in-storage. Replicated data is computationally straightforward and similar to working with purely local storage. However, erasure coded data requires an understanding of data element boundaries, as discussed in other sections of this disclosure (see, for example, Figure 32). These boundaries and how they are addressed vary from method to method, but in a naive striped environment they pose complex problems for both the target and the host. The target must be intelligent enough to identify data elements, and the host must be able to reconstruct any data element from its constituent parts.

[0273] Figure 32 shows one example of these challenges. Most data elements in a striping object cross boundaries, even though each one could easily fit within a single chunk. For example, Figure 32 shows an example of a striping object with multiple data elements that cross chunk and / or sub-block boundaries. In Figure 32, only the first data element (e.g., "Foo") is contained entirely within a single sub-block, but the remaining data elements cross chunk and / or sub-block boundaries and therefore cannot be easily manipulated within the target, but instead must be matched back to the host.

[0274] To solve this problem, the proposed solution provides a tool to slightly rearrange the data layout using padding to ensure that individual data elements reside neatly within their individual subblocks or chunks. In some cases, this may result in a small increase in storage overhead, but it greatly simplifies compute-in-storage operations as little coordination between the target and the host is required. A simple example of this layout is shown in Figure 33. For example, Figure 33 shows an example of a striping object with multiple data elements aligned within subblock or chunk boundaries using padding. In Figure 33, the layout of the striping object includes padding added to each data element to neatly align the data elements within the subblock and chunk boundaries. As a result, the complexity and required coordination between the target and the host can be significantly reduced.

[0275] To reduce the complexity of this compute-in-storage operation, we make a small change to the write path of the compute-in-storage application. The write path needs to be aware of the data layout of the target storage system, which is often trivially available or known out-of-band by a fixed policy (e.g., the Minio system stores the data layout of each object in a separate file that accompanies each object chunk). The writing application also needs to be able to identify the boundaries and sizes of individual data elements, and these data elements must be smaller than the chunk and sub-block sizes. After that, it becomes a simple greedy binning operation. This approach does not work if data elements are larger than the sub-block size, as these data elements will span sub-block and chunk boundaries regardless.

[0276] The following pseudocode shows an example algorithm for aligning data elements using padding: [Table 1]

[0277] If the host application or target can natively identify the boundaries, it can ignore the padding information for each sub-block, otherwise it can store this information out-of-band (e.g., in a separate file or database) or append it to the end of the sub-block stream.

[0278] If an application is not aware of padding, the host read path will need to be slightly modified, either by being aware of the boundaries and padding and being able to strip them, or by an alternate read path that has out-of-band knowledge of the path, which may be stored in the write path (as described above), appended to the file, stored in a separate database or metadata file, or stored as part of the extended filesystem metadata.

[0279] Typical compute-in-storage operations on the target data do not require additional modification beyond that required to enable the operation, such as recognizing the data elements.

[0280] Second, large results from block-based compute-in-storage operations need to be efficiently output. For example, a challenge with block-based approaches is that while small results can be easily embedded and returned using existing transport protocols (e.g., iSCSI bidirectional commands or NVMeOF fused commands), large results must be written to a storage target and retrieved by the host. In this case, storage must be allocated and zeroed from the host, adding expensive operations and thus eliminating many of the benefits of compute-in-storage. Naively trying to use methods such as fallocate means that there is no block mapping at the disk level until the data (or zeros) are actually written to disk, which is primarily a security issue.

[0281] To solve this problem, you need to preallocate a certain number of files on the host, and then use these files on demand as the output of a particular compute-in-storage operation. In some cases, you can use one or more preallocated files as a circular buffer and tell the target to overwrite the buffer as needed.

[0282] For preallocated file pools, after a compute-in-storage operation writes to a preallocated file, the file on the host needs to be truncated to reflect the new file length. We are assuming that the target returns the length of the output file. For example, if the file is compressed and the output is directed to one of the preallocated output files, the target will return the compressed file length to the host. From there, the host can truncate the preallocated file to the correct size. To truncate a preallocated file, the host's page cache must first be flushed (e.g., by unmounting and remounting the filesystem) or updated with the last page of the truncated file (e.g., by using a SCSI or NVMe pass-through command to read the file from disk and update the file on the host with a single write to the page cache). If the host page cache is not flushed or updated, stale data (e.g., data created during preallocation that still resides in the host page cache) may be written back to disk. This will corrupt the data output from the compute-in-storage operation on the target.

[0283] In some cases, rather than using a pre-allocated output file, the host may choose to use a pre-allocated circular buffer on the target. The compute-in-storage target only needs to maintain a very small amount of state information, such as the current "end" of the circular buffer, its initial address (the start), and the total length of the buffer. In other words, three integers. As computations are performed and results are written to the buffer, the target returns the offset and length of each result to the host and increments the current buffer end pointer. If writing a result would fill the buffer, the "end" pointer moves to the beginning and the old result is overwritten starting at the beginning of the space. This is noted in the reply to the host. For this reason, there must be enough buffer space available for the host to read the results before they are overwritten, depending on the host's policy. This basic concept is shown in Figure 34.

[0284] Figure 34 shows a simple example of the use of a preallocated circular buffer. It behaves as a typical circular buffer, but the only state maintained on the target side is the start, length and current end point of the buffer. As you continue writing, it will wrap around the space.

[0285] Below are some use cases for processing visual data using the proposed solution. The proposed solution is used for these use cases, but it should be noted that this list of use cases is not exhaustive and the proposed solution may also be used for any number of other use cases (visual and non-visual).

[0286] First, an overview of the Compute-in-Storage framework described in this disclosure is provided. In particular, the Compute-in-Storage framework relies on the concept of a virtual object. A virtual object is simply a temporary mapping that points to the location of a particular set of data, such as a file or object. In this case, it is a block storage object or an extent list that describes the location of logical blocks that the device can interpret. This mapping is part of a compute descriptor that describes the inputs, outputs, operations, and arguments of the Compute-in-Storage function. The Compute-in-Storage object interprets these compute descriptors and acts on the instructions contained within them. The result is either returned via a transport (e.g., iSCSI, NVMeOF, etc.) or written to a location on the storage object. This location can be detailed in the descriptor, or, if using a circular buffer or pre-allocated file approach, is returned to the host as an address and offset for subsequent reads and manipulations.

[0287] Image pre-processing is a use case. For example, a common set of operations on visual data is transforming an image, including operations like resizing and cropping. These tend to result in a smaller amount of data in the final image, but the size is still non-trivial and may not fit in the return. Furthermore, if a batch of images is stored together in a striped environment, they may cross sub-block boundaries in the manner described above.

[0288] With the proposed solution, it is easy to batch write images to striped storage: if one or more images are less than or equal to the sub-block size, simply apply the method described above to avoid images crossing sub-block boundaries.

[0289] The computational descriptor requires input of the image size and offsets (delimiters) to distinguish the boundaries of individual images, so that operations can be performed using common tools (e.g. OpenCV).

[0290] Very small results can be embedded directly into the return payload of the transfer (e.g. iSCSI, NVMeOF, etc.), while larger results can be written into the pre-allocated space mentioned above. Without such a methodology, the image would have to be returned in its entirety to the host, which would be extremely inefficient.

[0291] Image classification is another use case. For example, image classification involves running images through a trained machine learning model to identify what is in the image (e.g. cats or dogs). The same basic layout requirements mentioned above for image processing apply here as well (indeed, the result of image processing may be a precursor to classification). The computation descriptor describes the data boundaries of the image and how they are identified, and the result is written to a target location or returned directly as a payload.

[0292] Another use case is feature vector extraction. Feature vector extraction involves analyzing an image to extract features of the image or objects within it. There are various techniques for feature vector extraction, ranging from hand-selecting and extracting image features, such as a histogram of color intensity, to more complex extraction methods using intermediate layers of a convolutional neural network. In either case, the result is a vector of floating-point numbers describing the target elements that distinguish a particular visual object from others. The size (in bytes) of the extracted feature vector is typically fixed and orders of magnitude smaller than the input image, so it is unlikely that extra mechanisms will be needed to return the results. However, for use cases where feature extraction needs to be performed on a large number of images, the circular buffer technique described above can be used. However, since all feature vectors are of equal size, there is no need to calculate offsets.

[0293] Similar to image pre-processing, without the ability to store images in a single sub-block, intensive processing would be required on the host (e.g. reading multiple sub-blocks) negating most of the benefits of compute-in-storage. The read and write requirements are the same as for image processing, and the compute descriptors are similar: either an offset separating individual images or a defined delimiter is required.

[0294] Another use case is feature vector searching. For example, it is a common operation to search an existing database or table of feature vectors to identify which vectors are "similar" and use them for a target of a particular image. As with the image processing example, batch-stored feature vectors can be aligned within a fragment to facilitate this with little to no complex host-target coordination. Since the feature vectors corresponding to the multi-dimensional search space are all the same size, the boundaries of each feature vector can be statically determined, which is much easier than in the image case mentioned above (where the file size depends on the image content).

[0295] The result of this operation is usually a 2-tuple corresponding to the k nearest elements of the input feature vector.<distance, id> , and only a set of . If the number of results requested ('k') is small, they can be returned directly to the host. If 'k' is large, the offset and length of a pre-allocated buffer can be returned, and the host can do a normal read to get the results. However, the offset is simplified to be fixed, considering that each pair is a fixed size.

[0296] 35 shows a flowchart 3500 for performing visual computation tasks on image objects stored in a storage system. In various embodiments, the flowchart 3500 may be implemented using the embodiments and functionality described throughout this disclosure.

[0297] The flowchart begins at block 3502 where a request to write a set of image objects to a storage system is received.

[0298] The flowchart then proceeds to block 3504, where the set of image objects is divided into a number of chunks, each chunk of the number of chunks consisting of one or more image objects from the set of image objects, and the number of chunks is padded such that each image object from the set of image objects is aligned within the boundaries of a corresponding chunk of the number of chunks.

[0299] The flowchart then proceeds to block 3506, where the multiple chunks are written to the storage system, with each chunk being written to a corresponding one of the multiple storage devices.

[0300] The flowchart then proceeds to block 3508 where a request to perform a visual computation task on the set of image objects is received.

[0301] The flowchart then proceeds to block 3510, where it sends a plurality of computational offload commands to a plurality of storage devices, each computational offload command instructing a corresponding storage device to perform a visual computation task on a corresponding subset of image objects from the set of image objects, the corresponding subset of image objects being stored on the corresponding storage device.

[0302] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may restart at block 3502 to continue receiving and processing requests to write image objects to storage and to perform visual computation tasks on the image objects.

[0303] [Data aging in a distributed storage environment using compute-in-storage] In this section, we present a solution for aging data in a distributed storage environment using compute-in storage.

[0304] The ever-increasing amount of data stored from Internet-of-Things (IoT) sensors, audio / video devices, and countless Internet applications is putting pressure on available storage capacity. Typically, the value of data (and the corresponding access to it) declines after a period of time that is application-dependent. Strategies are therefore being implemented to "age" data to reduce the storage footprint. There are numerous approaches to aging. Old data can be compressed in-place or the encoding scheme can be changed to reduce the data footprint. Other approaches stem from the durability and availability guarantees offered by storage systems where multiple copies of individual data items are stored. Aging of replicated data can be done by erasure coding the data (more computation is required for encoding / erasure coding, but the storage footprint is much smaller). Finally, degraded data may be moved to a cheaper / slower storage tier (e.g., from SSD to HDD). In this case, the data is moved to the cheaper tier as is or compressed first. The ultimate goal of all these approaches is to reduce storage costs.

[0305] A separate trend is storage disaggregation, where storage is provisioned on blades with minimal processing power and minimal hardware resources. The storage media on the blades is exposed as block devices to a file system running on another server. This allows storage capacity to scale independently without other hardware resources being underutilized (or overpaid) as would be the case if a full fledged server were to house the storage.

[0306] Compute-in-storage is an emerging technology with broad applications. The solutions in this section address the need for new compute-in-storage mechanisms to efficiently "age" data (e.g., compress, tier, etc.) in the context of distributed storage systems (e.g., iSCSI, NVMeOF, etc.), such as rack-scale designs where host and storage servers are physically separated via a storage fabric.

[0307] Data aging requires reading the stored data, performing some transformation (e.g. compression), and storing it again. Once the storage is separated, the data needs to be read from the storage blade to the server running the OS or file system. Reading large amounts of old data consumes a lot of network bandwidth. For example, existing solutions require transferring the data from the storage blade to another server, where the data is transformed, and then written out again over the network to its final location. The T-10 standard defines an "XCopy" mechanism that allows direct transfer of data from one location (e.g. storage blade) to another. However, XCopy does not perform any operations that transform the data. Therefore, existing solutions consume significant network bandwidth both in transferring the original data and in writing the transformed data back to the storage layer.

[0308] Therefore, a compute-in-storage solution is proposed that allows aging of data on storage blades without loading all the data on the server running the file system and / or application. For example, instead of transferring data back and forth, the proposed solution sends a command to age the specified data. The command is much smaller than the data, so costly data transfers can be avoided. The command describes the operation and the list of blocks to operate on. In this way, aging can be done at the block layer without a file system or other storage stack (e.g., object storage system) running on the storage target.

[0309] This proposed solution has many advantages. For example, the proposed solution allows operations to be performed on the storage blades directly at the block layer, without the need for a file system or other storage stacks (e.g., object storage stack). As a result, less processing power is required on the storage blades. This increases the overall system efficiency and reduces the total cost of ownership (TCO) since aging operations can be performed in-situ.

[0310] The proposed solution also eliminates the need to transfer large amounts of data in the network just for aging. The reduced network bandwidth consumption for "background" storage operations reduces the perturbations caused by active "foreground" storage operations. Moreover, operators currently need to have a separate cluster network for such background traffic (which increases the total cost of deployment), whereas the proposed solution does not.

[0311] Additionally, the proposed solution may use more sophisticated aging transformations depending on the application. For example, data may initially be stored raw (e.g., for quick application access), but after some time, the data may be encoded to reduce the footprint. As another example, a video application may add metadata to identify frames that are not relevant to the application. In other cases, the video may have minimal changes during a time interval, as is often the case in security camera footage or during active play in sports recordings. This metadata may then be used on the storage blades during aging to cut out irrelevant frames entirely, resulting in a significant reduction in storage capacity.

[0312] An example of a computational architecture for aging data using compute-in-storage is shown in Figure 36. In particular, Figure 36 shows different system components (e.g., system stacks on the host and storage target machines) and the tasks each component performs to accomplish the aging operation.

[0313] An aging operation typically involves transforming data and placing it in an appropriate location in the storage system. In the proposed solution, data is not transferred from the block storage object for the aging transformation, but instead a command containing the appropriate information is sent to the storage object, allowing the storage object to perform the aging step itself.

[0314] This command specifies a list of blocks on which the aging transformation operation should be performed. The basic flow of such a command is further described in other sections of this disclosure. Effectively, the client works with the host to create a computational descriptor that describes the block locations of the file or object, along with the operations to be performed. However, in the case of aging, rather than returning the results of the operation, it transforms the data on the target side and returns metadata describing the location and length of the new data.

[0315] Several different approaches and embodiments for aging and migrating data using Compute-in Storage are presented below: (1) Aging of data to pre-allocated locations on the same target. (2) In-place data aging on the same target. (3) Aging of data to pre-allocated locations on different targets. (4) Aging the replicated data to erasure-coded data. (5) Aging of data to a third-party storage service.

[0316] Aging Data to a Pre-Allocated Location on the Same Target: The simplest method involves using a pre-allocated location on the same storage target where the transformed data is written. For example, a compression operation reads data specified in a compute descriptor and writes the compressed data to a target location also specified in the compute descriptor. Success / failure and data length are returned to the host / client, and the compressed data length is made available at the specified location for reading at a convenient time. Similarly, the old uncompressed data can be discarded when the host decides to do so. An example compute descriptor using this approach is shown in FIG. 37. In particular, FIG. 37 shows an example compute descriptor for compressing data in the manner described above and placing it in a new location.

[0317] Care must be taken when preallocating files, in particular that the files must be "synced" on the host before any compute-in-storage operations can be performed on them. If the files are not synced, the preallocated blocks will not have valid block mappings.

[0318] In-place data aging on the same target. Alternatively, a file can be compressed and written "in-place", i.e. truncating the original file and overwriting it with the compressed version. To truncate a file, the host's page cache must first be flushed (e.g. by unmounting and remounting the file system) or updated with the last page of the truncated file (e.g. by using a SCSI or NVMe pass-through command to read the file from disk and update the file on the host with a single write to the page cache). It should be noted that the "drop cache" standard command is advisory and that the file system may choose to ignore it. If the host page cache is not flushed or updated, stale data (e.g. data created during the preallocation phase and remaining in the host page cache) may be written back to disk. This would corrupt the data that is the output of the compute-in-storage operations on the target. The proposed solution may therefore implement various mechanisms to prevent stale data from being written out to storage.

[0319] Aging data to a pre-allocated location on another target: In this approach, the aging operation not only compresses the data, but also involves moving the compressed data to another target. The most common scenario is when the final target is also fabric partitioned, so the aging command requires information about the destination location in addition to the destination namespace block.

[0320] An example of such a command is shown in Figure 38. In particular, Figure 38 shows an example of a compute descriptor for aging data to a pre-allocated location on a different target machine. The input extent and length indicate the source block of data to age. The output address identifies the target machine where the aged data should be placed. The output extent and length provide information about the block on the target machine where the data will be stored. The method field indicates that the data should be gzip compressed before being sent to the target.

[0321] Figure 39 shows an example of a computation architecture for aging data to different targets using compute-in-storage. In Figure 39, the storage target that receives the computation descriptor not only needs to read the data and perform the aging operation, but also needs to call the block storage initiator with the output information of the computation descriptor.

[0322] Aging replicated data to erasure-coded data. In this approach, aging is achieved by converting replicated data to erasure-coded data. For example, data is often replicated to improve durability, high availability, and throughput of storage requests to the data. However, replication also increases the storage footprint. As data ages and becomes less valuable, one approach to reduce the storage footprint is to erasure code the data. Currently, aging in this manner requires reading one of the data copies into the host, erasure-coding the data, and finally writing out the erasure-coded chunks.

[0323] However, in the proposed solution, aging to erasure code the replicated data can be done in-situ using compute-in-storage. An example compute-in-store aging command for such an operation is shown in FIG. 40. In particular, FIG. 40 shows an example compute descriptor for converting replicated data to erasure coded data using the proposed solution. The compute descriptor includes erasure coding (EC) profile information such as the number of data chunks and the number of parity chunks. Additionally, the descriptor includes information on the storage targets where the various erasure coded chunks will be placed. If a successful response to the operation is received, the client application can update the metadata to indicate that the data has been aged and erasure coded with the chunks now located at the targets specified in the compute descriptor. Finally, the client application can send a command to delete the original replica of the data to free up storage space.

[0324] Aging data to a third-party storage service: Another possible approach is to age the data to another storage service (e.g., Ceph, Swift, Minio, etc.), possibly managed by another provider. Such an aging operation requires knowledge of the endpoint and authentication information of the third-party storage service. For example, Figure 41 shows an example of a computation descriptor for aging data to a third-party storage service. In this example, the computation descriptor identifies the data to be aged and further specifies that the data is to be transferred to the Minio storage service.

[0325] 42 illustrates a flowchart 4200 for performing a data aging calculation task on a data object stored in a storage system. In various embodiments, the flowchart 4200 may be implemented using the embodiments and functionality described throughout this disclosure.

[0326] The flowchart begins at block 4202 where a request is received to perform a data aging calculation task on a data object stored in a storage system, the data object being stored on a corresponding storage device of the storage system.

[0327] The flowchart then proceeds to block 4204, where a data aging calculation offload command is sent to a corresponding storage device, where the data aging calculation offload command instructs the corresponding storage device to perform a data aging calculation task on the data object.

[0328] The flowchart then proceeds to block 4206, where the results of the data aging computation offload command are received from the corresponding storage device.

[0329] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may restart at block 4202 to continue receiving and processing requests to perform data aging computation tasks on data objects stored in the striped storage system.

[0330] [NAND storage hint-based lifespan prediction] In this section, we present a solution that provides hint-based end-of-life prediction for NAND storage.

[0331] NAND performance is heavily influenced by how data is allocated within the device. Unlike traditional hard disk drives, NAND devices remap host writes to available pages within the device. These pages are organized into larger erase blocks. As data is overwritten, the old page in an erase block is invalidated and later made available for new data by garbage collection. Any data not yet overwritten in that erase block must then be relocated to another erase block on the same drive.

[0332] Because data reallocation is costly, it is desirable for all pages within an erase block to have the same lifespan (e.g., be deleted or overwritten by the host at roughly the same time), so that there is no need to reallocate data and the entire erase block can be efficiently reused for new data.

[0333] However, in many cases the lifespan of the data is unknown, making efficient garbage collection within the drive difficult.

[0334] Previous solutions rely on the operating system to send "hints" to the solid-state drive (SSD). In either case, the application must be modified to provide the hints. The biggest drawback of application-based hinting approaches is that the application must have some awareness of the lifetime of its data. In many cases, application developers simply do not know the lifetime of the data. Also, not all data is created by applications. Some data, such as file system metadata, has an associated lifetime that is useful to know for management of NAND storage.

[0335] In this section, we propose a solution that predicts and utilizes data lifetime when allocating NAND storage pages. Pages that are predicted to have the same lifetime are allocated to the same erase block.

[0336] The solution relies on I / O classification, where applications and file systems inform the block storage system of the "class" of data. These classes contain a variety of information, including the type of data, the process for reading / writing the data, and the intended use of the data. These "features" can be used to train machine learning models to predict longevity. For example, it may discover that all data classified as "metadata" is short-lived, while data classified as "archive" is long-lived.

[0337] At the core of this solution are two key aspects: first, it provides a mechanism by which applications can communicate classification information to the operating system and / or storage system; second, once classified, it utilizes a Classification and Regression Tree model (CART) to perform the actual predictions. Note that the mechanism for classifying data relies on "compute-in-storage", using a computational storage library to allow applications to communicate any classification information to the storage system.

[0338] This solution has many advantages: first, compute-in-storage does not require significant changes to the operating system (as opposed to other hinting approaches that rely on kernel patches); second, end-of-life prediction can help improve the performance of SSDs and / or any other type of NAND-based storage.

[0339] The proposed solution includes a computer system that includes at least one NAND storage device and one Optane device that acts as a cache for the NAND device. The cache device can be managed using Intel's Cache Acceleration Software (CAS), which also serves as an input source for I / O-related feature vectors to a machine learning (ML) algorithm. For example, a special API (e.g., Compute-in-Storage API) can be used to send classification information to the CAS, and a machine learning (ML) algorithm can be used to analyze the I / O trace data generated by the CAS. The ML algorithm is then trained to predict the lifetime of the data.

[0340] An example of an I / O feature vector used in the machine learning algorithm is shown in Table 1 below. [Table 2]

[0341] The proposed solution also periodically retrieves a small set of SSD counters, which are also provided as input feature vectors for the ML algorithm. These counters are read using the NVMe GetLog admin command. An example of the SSD feature vectors used by the machine learning algorithm is shown in Table 2 below. [Table 3]

[0342] Applications can provide additional feature vectors as input to the ML algorithm to help determine longevity. Applications pass these feature vectors to the CAS software using special APIs (e.g., the Compute-in-Storage API). Examples of application feature vectors used by the machine learning algorithm are shown in Table 3 below. [Table 4]

[0343] Figure 43 shows an example of the architecture and data flow for hint-based end of life prediction of NAND storage.

[0344] To illustrate, an end-to-end use case in the context of scale-out multi-tenant cloud storage (e.g., distributed storage shared among many tenants with different I / O characteristics) is shown in FIG. 44. For example, to realize longevity prediction as an end-to-end pipeline in a multi-tenant environment (e.g., using Red Hat Ceph), FIG. 44 shows an exemplary embodiment of end-to-end I / O classification and computation offloading descriptor mapping. This example leverages two types of I / O: metadata and data. For example, as described below, the mechanism shown in FIG. 44 has two components:

[0345] The first component is the I / O classification extension of the compute offload descriptor, which provides application level I / O classification along the end-to-end path to the final SSD storage. Any intermediate layer can be a consumer of this information for longevity prediction at that layer. For example, CAS can be used at the QEMU Ceph RBD level to cache data on the QEMU host.

[0346] The second component is the mapping and translation of compute offload descriptors. Sequential or random I / O operations seen by the application side may not necessarily be seen as sequential or random at various tiers along the end-to-end I / O path. Therefore, the descriptors need to be appropriately remapped and translated so that the I / O classification information is always associated with the correct I / O for that tier. For example, as shown in Figure 44, guest Virtual Machine (VM) file system metadata operations on a Ceph cluster-backed virtual disk image need to be translated to the appropriate descriptor format for the corresponding tier.

[0347] For example, the I / O classes of "meta" and "data" are added to the compute offload descriptor as shown in Figure 44. The I / O classification information originates within the cloud tenant's VM instance and is tunneled through all intermediate layers such as QEMU (Virtual Machine Manager), Ceph RBD (Block to Object Memory Mapping Service), Ceph Protocol (Ceph Memory Service Protocol over TCP / IP), Ceph OSD (Object Memory Daemon), NVMeOF (Distributed Memory Fabric), memory targets, and finally to the underlying storage (SSD or HDD).

[0348] Lifespan prediction can occur at any tier along the end-to-end I / O classification path, depending on which tier is designed to be the consumer of I / O classification. For example, if the goal is to deploy an Optane cache device on the QEMU VMM host side, QEMU Ceph RBD will be the consumer of I / O hints and will predict lifespan using the feature vectors shown in Tables 1 to 3.

[0349] 45 shows a flowchart 4500 for storing data in NAND storage using data age prediction. In various embodiments, flowchart 4500 may be implemented using the embodiments and functionality described throughout this disclosure.

[0350] The flowchart begins at block 4502 where a write command is received from a host processor, the write command instructing the NAND storage device to perform a write operation to write a data object to the NAND storage device.

[0351] The flowchart then proceeds to block 4504, where it identifies a number of input / output (I / O) characteristics associated with the write operation, the number of I / O characteristics indicating a timestamp, a type of data, and an application name.

[0352] The flowchart then proceeds to block 4506, where the machine learning model is trained to predict the lifespan of the data object based on a plurality of I / O characteristics.

[0353] The flowchart then proceeds to block 4508, where a corresponding block on the NAND storage device is selected for storing the data object, the corresponding block being selected from a plurality of blocks on the NAND storage device based on the lifetime of the data object.

[0354] The flowchart then proceeds to block 4510, which illustrates writing the data object to a corresponding block on the NAND storage device.

[0355] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may restart at block 4502 to continue receiving and processing requests to write data objects to NAND storage.

[0356] [A heuristic method for efficient search of cache allocation policies] In this section, we present a solution for heuristically exploring cache allocation policies.

[0357] Storage analytics is a research area aimed at automating routine storage management tasks, such as cache configuration and provisioning, through automated workload monitoring (e.g., I / O characterization), policy optimization (e.g., cache modeling and prediction), and orchestration (e.g., bringing caches online with optimized policies).

[0358] Consider the process of properly configuring your caching software (e.g., Intel's Cache Acceleration Software (CAS), which supports Differentiated Storage Services). In such a system, each class of data can receive its own caching policy. For example, you might choose to cache the file system journal but not cache large files.

[0359] However, even for a relatively small number of I / O classes, the number of possible cache allocation policies grows exponentially. It is not feasible to naively search this configuration space to find the optimal caching policy. Instead, we must rely on heuristics and models to predict the cache performance of different cache policies.

[0360] In particular, cache software typically performs a process to automatically configure itself, a process known as a "cache utility." For example, Intel's CAS (Cache Acceleration Software) performs the following process to automatically configure itself (as shown in Figure 46): Stage 1-2 You can run an application and trace its I / O, including the class of each I / O in the trace. Examples of I / O classes include "metadata", "small file", "medium file", and "large file". Stage 3-4 Configure the CAS cache emulator with a "cache everything" policy using an infinite cache size. No I / O is ever evicted. Everything is cached in the emulator. Replay the trace through the emulator and within the emulator record the number of disk seeks and the number of cache lines consumed for each I / O class. Dividing the former by the latter gives the cache utility for that class. In other words, if we had an infinite cache we could cache all data and determine the size of each class, and then determine how many disk seeks we could save by caching the entire class. Call this the "cache utility" for that class. Higher is better. Utility(class) = seeks(class) / size(class). Steps 3-4 (Determine eviction policy). Rank all n classes from most to least useful. The rank represents the eviction priority for each class of I / O. That is, when the cache is full, evict classes with low utility first, then classes with low utility, and so on. Classes with high utility are preferred in the cache. Steps 5-6 (Determine Allocation Policy): Based on the eviction policies for the n classes found in the previous step, CAS needs to decide which classes to allocate. Not all classes will necessarily be allocated to the cache. To achieve this, CAS starts by caching only the highest priority class and records the cache performance (by emulation). It then adds the second highest priority, then the third, and so on until finally all classes are cached. By doing so, CAS can figure out when the cache performance has reached its maximum and use that as the cache allocation policy. In many cases, it is necessary to cache only a certain number of classes to get the maximum performance. After that, it may progress to "thrashing". Note that this step is linear in the number of classes. Each time you add a new class to the cache, you need to run the cache emulator once. Phase 7 Communicate eviction rankings and allocation policies (which classes are cached) to the actual CAS software (not a CAS emulator), allowing it to set appropriate caching policies at runtime.

[0361] The processing time for allocation tuning is inefficient: it scales linearly with the number of classes and also with the size of the workload (e.g., replaying a trace of a given workload through an emulator takes time). A solution that requires fewer emulator invocations is desirable.

[0362] Therefore, in this section, we describe how to automatically explore the cache allocation space (e.g., which classes of data should or should not be allocated in the cache). In particular, the proposed solution is a novel automatic cache allocation tuning technique that combines hill climbing and modeling methods. For example, we use the minimum number of runs required to start fitting polynomial, sigmoid, and exponential models. We then perform a hill climbing search to determine which model has the smallest residual, and use that model to predict the next optimized allocation policy while dropping the remaining models. When the residual of the model is near zero (~0), we stop the search. The model obtained in this way becomes the final model, and the predicted optimal allocation policy becomes the optimized allocation configuration. In this way, the cache is automatically optimized and tuned to maximize performance using the proposed solution.

[0363] The proposed solution consists of eight stages to automatically set the CAS allocation policy. It is illustrated in detail using the example of the Postmark workload (modeling SPEC SFS). The nine CAS classes and their eviction policies, determined by the cache utility (as mentioned above), are shown in Table 5. In the emulated runs, this utility-based eviction policy is never changed; instead, it focuses only on exploring the cache allocation space.

[0364] In contrast to the linear solution mentioned above (e.g., looking at the effect of allocating each class at once), instead we do something like a binary search: start by caching only the first class (the one with the most utility) and record the emulated cache performance. Then cache all classes and record that performance too. We can then fit a line to these two points and extrapolate / interpolate between them to determine the effectiveness of caching other combinations. But instead of stopping there, find a "middle" point by caching only half the classes. Now we have three points to which we can fit a curve - not necessarily a straight line; instead, find the best fit (e.g., linear, quadratic, cubic). We keep doing this until we find the best fitting curve. We then use that curve to predict the results of caching other classes of data.

[0365] More details of this process are given in Table 6, showing the number of loops and tests / loops required to find the optimal cache allocation settings. Phase 1 of the proposed solution runs three allocation emulation tests with associated eviction order and I / O class at two boundaries and a center point. For the Postmark workload, three emulation tests are run with allocations I / O class = Metadata (boundary 1), I / O class = All (boundary 2), and I / O classes = 1, 11, 12, 13, and 14 (center point). Phase 2 fits a fitable model using three data points from Phase 2. The minimum data points required to fit various models are given in Table 4. Linear, Quadratic, Logistic 3P, and Exponential 3P are fit with three data points. [Table 5]

[0366] Since a particular workload has a unique combination of I / O classes, the boundary conditions are bounded. There is only one optimal model that can pass through the points inside the boundary. Thus, we are looking for a perfect fitting model for the bounded boundary problem based on the residuals. In stage 3, we calculate the average residual between the predicted and actual values ​​and use this to perform a hill climbing search to find the best fitting model.

number

number

[0367] In the first model fitting with three data points, a hill-climbing search is performed to find the model with the smallest mean residual of the three data points, and this model is used to predict the next optimized allocation class in step 4. In step 5, if the predicted allocation class is not among the first three data points, emulation is performed with the new predicted allocation setting. As shown in Table 6, the mean residual of the quadratic model is approximately zero (~0), but the optimized allocation configuration is predicted to include allocations of classes 1, 11, 12, 13, 14, 15, and 16 that have not been tested so far. Then, emulation is performed with the new predicted optimized configuration and loop 2 is continued. Model fitting continues with the previous best model and adds a new model that can be fitted with four data points in step 6. The models in loop 2 include quadratic, cubic, and logistic 4P. In step 7, steps 3 to 6 are repeated until a model is found that satisfies the conditions that the mean residual is approximately zero (~0) and its predicted optimized setting is one of the previous measured data points. This phenomenon occurred in loop 3 of the Postmark workload. The fourth-order model predicted that the assignment to classes 1, 11, 12, 13, and 14 was optimal, with an average residual of approximately zero (~0). This predicted configuration was then executed in loop 1. Thus, the loop stops and in stage 8, the predicted optimum by the optimal model is selected as the optimal configuration for the workload. [Table 6] [Table 7]

[0368] In summary, the allocation policy lookup loop is as follows: (1) Once the eviction rank is known, first perform three emulations (two points from the boundary + one point at the center). (2) Fit all models requiring a minimum of three points (m). (3) Calculate and compare the average residuals of the models to find the minimum value. (4) The model with the smallest residual error is used to predict the optimal allocation setting. (5) If the predicted value is not included in the three points in step 1, perform emulation. (6) Fit a model requiring a minimum of 4 points (m+1) and refit the best model found earlier in step 3. (7) Repeat steps 3–6 until the average residual is near zero (~0) and the predicted optimal allocation setting is among the runs tested. (8) Update the predicted allocation policy of the best model to CAS.

[0369] 47 shows a flowchart 4700 for optimizing and configuring a cache in a storage system. In various embodiments, the flowchart 4700 may be implemented using the embodiments and functionality described throughout this disclosure.

[0370] The flowchart begins at block 4702 with identifying a cache eviction policy for a set of dataclasses, where the cache eviction policy indicates a priority for evicting each dataclass in the set of dataclasses from the cache.

[0371] The flowchart then proceeds to block 4704, where it identifies a set of cache allocation policies to evaluate based on the cache eviction policy, where each cache allocation policy is to allocate a different subset of the set of dataclasses to the cache.

[0372] The flowchart then proceeds to block 4706, where performance of the cache for a first subset of cache allocation policies from the set of cache allocation policies is evaluated, and a first set of performance metrics is obtained for the first subset of cache allocation policies based on evaluating the performance of the cache.

[0373] The flowchart then proceeds to block 4708, where a plurality of curve-fitting models are evaluated for the first set of performance metrics.

[0374] The flowchart then proceeds to block 4710 and identifies an optimal curve fitting model for the first set of performance indicators, where the optimal curve fitting model is identified from the multiple curve fitting models evaluated with the first set of performance indicators.

[0375] The flowchart then proceeds to block 4712, where cache performance is predicted for a second subset of cache allocation policies from the set of cache allocation policies, where a second set of performance metrics is predicted for the second subset of cache allocation policies based on the best fit curve fitting model for the first set of performance metrics.

[0376] The flowchart then proceeds to block 4714, where an optimal cache allocation policy is selected from the set of cache allocation policies, where the optimal cache allocation policy is selected based on the first set of performance indicators and the second set of performance indicators.

[0377] The flowchart then proceeds to block 4716, where the cache is configured based on a cache eviction policy and an optimal cache allocation policy.

[0378] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may restart at block 4702 to continue optimizing and configuring the cache.

[0379] [CRC calculation using computation memory] This section describes a compute-in-storage solution for performing end-to-end CRC calculations in SSDs on one or more Logical Block Addresses (LBAs).

[0380] Advances in storage technology have led to the development of very fast devices. Various forms of non-volatile memory, including NAND and persistent memory, are used, especially for solid-state disks. However, I / O speeds have not kept up, and data transfer between storage and the CPU is becoming slower. The available bandwidth for these high-speed devices is managed by an I / O stack that includes various legacy software and hardware. Currently, improvements to the I / O path are being made, but they do not solve the fundamental problem of data taking a long time to move. Non-volatile memory is already limited by the I / O controllers attached to it, and this is expected to get worse as storage density increases.

[0381] This solution proposes a compute-in-storage alternative to SSDs: instead of moving the data to the CPU, we want to move the computation to the data. We focus specifically on CRC calculations, which are data-intensive and commonly used in various software stacks to ensure data integrity.

[0382] Specifically, it calculates the CRC for one or more LBAs in the SSD and returns the result to the host without transferring the data payload, allowing the host to offload data integrity checking to the storage device rather than transferring the entire payload and then performing the data integrity calculation.

[0383] As a concrete example, consider "bit rot detection" or "auditing" in object storage systems, which identifies corruption and triggers subsequent repair by calculating a checksum (often CRC-32) over the entire object. The problem is that as drives and storage systems proliferate, the amount of bandwidth required to read the data and calculate checksums continues to grow, overwhelming the I / O path with "uninteresting" and important traffic. If the checksums can be offloaded to the SSD itself, this frees up CPU and I / O bandwidth for higher priority, latency-sensitive computations.

[0384] Current solutions require the host to transfer the entire data payload over the transport interface and use the host CPU to perform the data integrity calculation. Thus, both the host CPU and the transport interface bandwidth are required resources to perform the data integrity check. If the CRC calculation could be offloaded to the storage device, the host CPU and transport interface bandwidth could be used for more valuable work, but this is not possible with current solutions.

[0385] In this section, we propose a compute-in-storage solution that allows offloading the CRC calculation to the storage device. For example, in the transfer hardware, a new feature is provided where a DMA operation accumulates the CRC on a payload of ≥ 1 sector and returns the calculated CRC to the host in response to a vendor-specific command. The vendor-specific command requests the CRC calculation on a payload of one or more 4K sectors and further includes a final length that allows the calculation on data that is not a multiple of the sector length in size. That is, the final sector length may be less than 4K and the final CRC update is calculated on a subset of that data. The firmware in the SSD interprets this command and reads the sector from the NAND storage. A DMA descriptor is programmed into the transfer hardware, which instructs the transfer hardware to read the data and accumulate the CRC on the payload, but to discard the data rather than forwarding it to the host memory via the corresponding interface (e.g., SATA, PCIe, etc.). After all sectors have passed through the transfer hardware, the final CRC value is returned to the host. FIG. 48 illustrates a flowchart of an example embodiment of offloading CTC computation using Compute-in-Storage, as described above.

[0386] With this solution, the transfer hardware provides the ability to accumulate a CRC for >=1 LBA as described in the DMA descriptor. After the CRC calculation is done, the data payload is discarded and no data is transferred to the host until all LBAs have been processed and the final CRC value returned.

[0387] By implementing vendor-specific commands to offload CRC checking to the storage device, a user and / or entity can free up the host's CPU for higher value tasks, improving overall performance.

[0388] 49 shows a flowchart 4900 for performing CRC verification on a data object stored in a storage system. In various embodiments, the flowchart 4900 may be implemented using the embodiments and functionality described throughout this disclosure.

[0389] The flowchart begins at block 4902 with receiving a request to perform a cyclic redundancy check (CRC) verification on a data object stored in a storage system, the data object being stored on a corresponding storage device in the storage system.

[0390] The flowchart then proceeds to block 4904, which illustrates sending a CRC calculation offload command to the corresponding storage device, where the CRC calculation offload command instructs the corresponding storage device to perform a CRC verification on the data object.

[0391] The flowchart then proceeds to block 4906 where the results of the CRC calculation offload command are received from the corresponding storage device.

[0392] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may resume at block 4902 to continue receiving and processing requests to perform CRC verification on data objects stored in the storage system.

[0393] [Example of calculation implementation] The following sections provide various examples of computing devices, systems, architectures, and environments that can be used in conjunction with the compute-in-storage functionality described throughout this disclosure.

[0394] Edge Compute Architecture Examples The following embodiments relate generally to implementing data processing, service management, resource allocation, compute management, network communications, application partitioning, and communications systems, and more particularly to techniques and configurations for adapting various edge computing devices and entities to dynamically support multiple entities (e.g., multiple tenants, users, stakeholders, service instances, applications, etc.) in a distributed edge computing environment.

[0395] In the following description, methods, configurations, and associated apparatus are disclosed for various improvements to the configuration and functional capabilities of edge computing architectures and implemented edge computing systems. These improvements are useful for a variety of use cases, particularly those involving multiple stakeholders in the edge computing system, such as multiple users of the system, multiple tenants on the system, multiple devices or user equipment interacting with the system, multiple services provided by the system, multiple resources available or managed within the system, multiple network access forms exposed by the system, multiple operational locations of the system, etc. Such multi-dimensional aspects and considerations are generally referred to herein as "multi-entity" constraints, and specifically describe resources managed or orchestrated in a multi-tenant and multi-service edge computing configuration.

[0396] Exemplary edge networking systems described below move computational and storage resources closer to the edge of the network (e.g., closer to clients, endpoint devices, or "things"). By moving computational and storage resources closer to the devices that generate or use data, various latency, compliance, and / or cost constraints can be achieved compared to standard network (e.g., cloud computing) systems. To that end, in some examples, a pool of computational, memory, and / or storage resources is located and / or otherwise equipped on local servers, routers, or other network equipment. Such local resources help meet the constraints imposed on the system. For example, utilizing local computational and storage resources allows edge systems to perform computations in or near real-time. This may be a consideration for low-latency user cases such as autonomous driving, video surveillance, and mobile media consumption. Additionally, these resources can benefit from service management at the edge system. This provides the ability to extend and achieve local SLAs, manage tiered service requirements, and enable local features and functionality, either temporarily or permanently.

[0397] An exemplary edge computing system may support and / or provide various services to endpoint devices (e.g., client user equipment (UE)), each of which may have different requirements or constraints. For example, some services may have priority or quality of service (QoS) constraints (e.g., traffic data from an autonomous vehicle has a higher priority than temperature sensor data), reliability or resilience constraints (e.g., traffic data requires mission-critical reliability, while temperature data can tolerate some error variability), and even power, cooling, and form factor constraints. Such technical constraints, when applied in a multi-stakeholder environment, can become very complex and technically challenging.

[0398] As used herein, the term "edge computing" may encompass many distributed computing implementations that move processing activities and resources (e.g., computation, storage, acceleration resources, etc.) to the "edge" of a network to reduce latency and improve throughput for end-point users (e.g., client devices, user equipment, etc.). Such edge computing implementations typically include providing such activities and resources in a cloud-like service, function, application, or subsystem from one or more locations accessible via a wireless network. Thus, references to the "edge" of a network, cluster, domain, system, or computing arrangement as used herein may refer to a group or grouping of functional distributed computing elements and thus are generally unrelated to "edges" (links or connections) as used in graph theory.

[0399] The specific arrangement of edge computing applications and services accessible over mobile wireless networks (e.g., cellular and Wi-Fi data networks) is referred to as "Mobile Edge Computing" or "Multi-Access Edge Computing" and sometimes referred to by the acronym "MEC." The term "MEC" herein may also refer to the standardized implementation published by the European Telecommunications Standards Institute (ETSI), referred to as "ETSI MEC." Terms used in the ETSI MEC specifications are generally incorporated herein by reference unless a contrary definition or usage is provided herein.

[0400] As used herein, the term "computing node" or "computing device" refers to an identifiable entity that implements one aspect of the operation of edge computing, whether it is part of a larger system, a collection of distributed systems, or an independent device. In some examples, a computing node may be referred to as an "edge node," "edge device," or "edge system," whether it is operating as a client, server, or intermediate entity. Particular implementations of a computing node may be incorporated into a server, base station, gateway, roadside unit, on-premise unit, UE or end consumption device, etc.

[0401] As used herein, the term "base station" may refer to a network element in a radio access network (RAN), such as a fourth generation (4G) or fifth generation (5G) mobile communications network, responsible for transmitting and receiving radio signals in one or more cells to and from user equipment (UE). A base station may have built-in antennas or may be connected to an antenna array by a feeder cable. A base station may use dedicated digital signal processing and network function hardware. In some examples, a base station may be divided into multiple software-operated functional blocks for resiliency, cost, and performance. In some examples, a base station may include an evolved Node-B (eNB) or next generation Node-B (gNB). In some examples, a base station may operate or include computational hardware to operate as a computational node. However, in many of the scenarios described herein, a RAN base station may be substituted with an access point (e.g., a wireless network access point) or other network access hardware.

[0402] As used herein, the term "Central Office" (or CO) means an aggregation point of telecommunications infrastructure within an accessible or defined geographic area, often referring to a location where a communications service provider traditionally places one or more types of access network switching equipment. A CO may be physically designed to house telecommunications infrastructure equipment as well as computing, data storage, and network resources. However, a CO need not be a telecommunications carrier designated location. A CO may host any number of computing devices for edge applications and services, and may also implement cloud-like services locally.

[0403] As used herein, the term "Cloud Service Provider" (or CSP) may refer to an organization that operates a typically large-scale "cloud" resource consisting of central, regional, and edge data centers (e.g., when used in the context of a public cloud). In other instances, a CSP may be referred to as a CSO (Cloud Service Operator). The term "cloud computing" generally refers to computing resources and services that a CSP or CSO provides in remote locations that at least have increased latency, distance, or constraints compared to edge computing.

[0404] As used herein, the term "data center" or "datacenter" refers to a structure designed to house multiple high-performance computing and data storage nodes, implying a large amount of computing, data storage, and network resources in one location. This often requires specialized rack or enclosure systems, appropriate heating, cooling, ventilation, security, fire suppression, and power supply systems, etc. The term may also refer to the computing and data storage nodes, depending on the context. Datacenters may vary in size between centralized or cloud datacenters (e.g., the largest), regional datacenters, and edge datacenters (e.g., the smallest).

[0405] The term "access edge tier" here refers to the sub-tier of the infrastructure edge that is closest to the end users and devices. For example, such a tier can be realized in edge data centers deployed at cellular network sites. The access edge tier acts as the frontline of the infrastructure edge and may connect to higher-level aggregation edge tiers.

[0406] As used herein, the term "aggregation edge tier" refers to a tier of infrastructure edge that is one hop away from the access edge tier. This tier can exist as a mid-sized data center in a single location or can consist of multiple micro data centers interconnected to form a hierarchical topology with the access edge to provide greater collaboration, workload failover, and scalability than the access edge alone.

[0407] The term "Network Functions Virtualization" (NFV) here refers to the migration of network functions from embedded services in dedicated hardware appliances to software-based Virtualized Network Functions (or VNFs) running on standardized CPUs (e.g., in standard x86 and ARM servers such as Intel® Xeon™, AMD® Epyc™, and Opteron® processors) using industry-standard virtualization and cloud computing technologies. In some aspects, NFV processing and data storage occurs at the infrastructure edge, in edge data centers directly connected to local cell sites.

[0408] As used herein, the term "Virtualized Network Function" (or VNF) refers to a software-based network function that runs on multifunction, multipurpose computing resources (e.g., x86, ARM processing architectures, etc.) that NFV uses in place of dedicated physical equipment. In some aspects, multiple VNFs run in edge data centers at the edge of the infrastructure.

[0409] The term "edge computing node" as used herein refers to a real-world, logical, or virtualized implementation of a computable element in the form of a device, gateway, bridge, system, subsystem, component, etc., whether operating in server, client, endpoint, or peer mode, and whether located at the "edge" of the network or at a more connected location within the network. "Node" as used herein is generally interchangeable with "device," "component," and "subsystem." However, an "edge computing system" generally refers to a distributed architecture, organization, or collection of multiple nodes or devices organized to achieve or provide some aspect of a service or resource in an edge computing environment.

[0410] As used herein, the term "cluster" refers to a set or group of entities as part of an edge computing system (or systems), which may be in the form of physical entities (e.g., different computing systems, networks or groups of networks, etc.), logical entities (e.g., applications, functions, security constructs, containers, etc.), etc. In some places, a "cluster" is also referred to as a "group" or a "domain." The membership of a cluster may change or be affected based on conditions or functions, including dynamic or property-based membership, network or system management scenarios, or various example techniques described below for adding, modifying, or removing entities in a cluster. Additionally, a cluster may include or have associated with multiple tiers, levels, or properties, including variations in security functions or results based on such tiers, levels, or properties.

[0411] While many of the following examples are provided using specific cellular / mobile network terminology, including the use of 4G / 5G 3GPP network components (or the anticipated terahertz-based 6G / 6G+ technologies), it will be understood that these examples are applicable to many other deployments of wide area and local wireless networks, as well as the integration of wired networks (including optical networks and associated fiber, transceivers, etc.).

[0412] FIG. 50 is a block diagram 5000 outlining an arrangement for edge computing, which includes a layer of processing referred to in many of the present examples as an “edge cloud.” As shown, the edge cloud 5010 is collocated at an edge location, such as a base station 5040, local processing hub 5050, or central office 5020, and thus may include multiple entities, devices, and equipment instances. The edge cloud 5010 is located much closer to endpoint (consumer and producer) data sources 5060 (e.g., autonomous vehicles 5061, user equipment 5062, business and industrial equipment 5063, video capture devices 5064, drones 5065, smart city and building devices 5066, sensors and IoT devices 5067, etc.) than the cloud data center 5030. The compute, memory, and storage resources provided at the edge of the edge cloud 5010 are critical for providing ultra-low latency response times to the services and functions used by the endpoint data sources 5060, and also provide benefits such as reducing network backhaul traffic from the edge cloud 5010 to the cloud data center 5030, thereby improving energy consumption and overall network utilization.

[0413] Compute, memory, and storage are scarce resources that typically decrease with edge location (e.g., consumer endpoint devices have less processing resources available compared to base stations or central offices). However, the closer the edge location is to the endpoint (e.g., UE), the more space and power are limited. Thus, edge computing, as a general design principle, seeks to minimize the amount of resources required for a network service by distributing more resources that are located closer together, both geographically and in terms of network access times.

[0414] Below, we describe aspects of edge cloud architectures that encompass multiple deployment possibilities and address constraints that network operators and service providers may have with their own infrastructure, such as configuration variations based on edge location (e.g., a base station-level edge may have more constrained performance), configuration based on the type of resources (e.g., compute, memory, storage, fabric, acceleration) available at an edge location, hierarchy of locations, or group of locations, services, security, management, orchestration capabilities, and associated goals for enabling ease of use and performance of end services.

[0415] Edge computing is an evolving paradigm of computing closer to the "edge" of the network, typically using compute platforms implemented in locations such as base stations, gateways, and network routers, closer to the endpoint devices that generate and consume data. For example, an edge gateway server may have a pool of memory and storage resources and perform computations in real time for low-latency use cases of connected client devices (e.g., autonomous driving and video surveillance). It is also possible to add compute and acceleration resources to base stations, for example, to directly process service workloads of connected user equipment without further communication data and without going through a backhaul network. As another example, the network management hardware in the central office may be replaced with compute hardware that performs virtualized network functions and provides compute resources for the execution of services and consumer functions for connected devices. In edge computing networks, scenarios are possible where compute resources are "moved" to data, or services where data is "moved" to compute resources. For example, the compute, acceleration, and network resources of a base station may provide services that scale as needed to meet the demands of the workload. This is to enable dormant capacity (subscription, capacity on demand) to address corner cases or emergency situations, or to significantly extend the implementation lifecycle of deployed resources. As another example, the compute-in storage capabilities described throughout this disclosure may be leveraged across clouds 5030, edge clouds 5010, and / or endpoints 5060 to reduce latency associated with storing, retrieving, and computing data.

[0416] In contrast to the network architecture in Figure 50, traditional endpoint (e.g., UE, vehicle-to-vehicle (V2V), vehicle-to-object (V2X), etc.) applications rely on local device or remote cloud data storage and processing to exchange and coordinate information. Cloud data placement enables long-term data aggregation and storage, but is not optimal for time-varying data such as collisions or signal changes, and may fail when trying to meet latency challenges.

[0417] Depending on the real-time requirements in the communication context, a hierarchy of data processing and storage nodes can be defined in an edge computing deployment. For example, such a deployment may include local ultra-low latency processing, regional storage and processing, and remote cloud data centric based storage and processing. KPIs (Key Performance Indicators) may be used to identify where sensor data should best be forwarded and where it should be processed and stored. This typically depends on the ISO layer dependency of the data. For example, data at lower layers (e.g. PHY, MAC, routing) is typically fast changing and is better processed locally to meet latency requirements. Data at higher layers, such as application layer data, is typically less time sensitive and may be stored and processed at remote cloud data centric locations.

[0418] FIG. 51 illustrates the deployment and orchestration of a virtual edge configuration in an edge computing system operated across multiple edge nodes and multiple tenants. Specifically, FIG. 51 depicts the coordination of a first edge node 5122 and a second edge node 5124 in an edge computing system 5100 to satisfy requests and responses to various client endpoints 5110 from various virtual edge instances. The virtual edge instances provide edge compute capabilities and processing in an edge cloud and access a cloud / data center 5140 for high latency requests such as websites, applications, database servers, etc. In this manner, the edge cloud can coordinate processing across multiple edge nodes for multiple tenants or entities.

[0419] In the example of FIG. 51, these virtual edge instances include a first virtual edge 5132 provided to a first tenant (tenant 1) providing a first combination of edge storage, compute, and services, and a second virtual edge 5134 provided to a second tenant (tenant 2) providing a second combination of edge storage, compute, and services. The virtual edge instances 5132, 5134 are distributed across edge nodes 5122, 5124, which may include scenarios where requests and responses are fulfilled from the same edge node or different edge nodes. Configuring each edge node 5122, 5124 to operate in a distributed yet cooperative manner is performed based on the edge provisioning function 5150. The ability of the edge nodes 5122, 5124 to provide coordinated operation of applications and services among multiple tenants occurs based on the orchestration function 5160.

[0420] It should be understood that among the devices of 5110, there are multi-tenant devices where Tenant 1 operates within a Tenant 1 "slice" while Tenant 2 operates within a Tenant 2 slice. Additionally, a trusted multi-tenant device may contain tenant-specific cryptographic keys, and the combination of key and slice may be considered as a "root of trust" (RoT) or a tenant-specific RoT. The RoT may further be dynamically configured and computed using a security architecture such as the DICE (Device Identity Composition Engine) architecture, which uses DICE hardware building blocks to build a layer-trusted computation-based context for stacking device functions (such as FPGAs (field programmable gate arrays)). The RoT may also be used as a trusted computation context to support each tenant operation.

[0421] Edge compute nodes may partition resources (memory, CPU, GPU, interrupt controller, I / O controller, memory controller, bus controller, etc.), each partition may include RoT functions, and fan-out and stacking may be further applied to edge nodes by the DICE model. Cloud compute nodes, consisting of containers, FaaS (Function as a Service) engines, servlets, servers, or other compute abstractions, may be partitioned according to the stacking and fan-out structure of DICE to support their respective RoT contexts. Thus, each RoT spanning entity 5110, 5122, and 5140 may coordinate the establishment of a distributed trusted computation base (DTCB) such that a tenant-specific virtual trusted and secure channel linking all elements end-to-end can be established.

[0422] Additionally, edge compute systems can be extended to provide orchestration of multiple applications using containers (contained, deployable units of software that provide code and required dependencies) in a multi-owner, multi-tenant environment. A multi-tenant orchestrator may be used to perform key management, trust anchor management, and other security functions related to the provisioning and lifecycle of the trusted "slice" concept in Figure 51. The orchestrator can create tenant-specific RoT contexts using the DICE stacking and fan-out structure. Thus, the orchestration capabilities provided by the orchestrator can participate as tenant-specific orchestration providers.

[0423] Thus, an edge computing system may be configured to satisfy requests and responses to various client endpoints from multiple virtual edge instances (and from cloud or remote data centers not shown). The use of these virtual edge instances simultaneously supports multiple tenants and multiple applications (e.g., AR / VR, enterprise applications, content delivery, gaming, compute offload, etc.). Furthermore, within a virtual edge instance, there may be multiple types of applications (e.g., regular applications, latency sensitive applications, latency critical applications, user plane applications, networking applications, etc.). Additionally, the virtual edge instances may span multiple owners' systems in different geographic locations.

[0424] In a further example, an edge computing system may deploy containers. As a simplified example, a container manager is adapted to run and launch instances of containerized pods, functions, and functions-as-services via compute nodes, or to run containerized virtualized network functions via compute nodes separately. This arrangement can be adapted to a multi-tenant system configuration, where the containerized pods, functions, and functions-as-services instances are launched within virtual machines specific to each tenant (aside from running virtualized network functions).

[0425] Within the edge cloud, a first edge node 5122 (e.g., operated by a first owner) and a second edge node 5124 (e.g., operated by a second owner) may operate or respond to a container orchestrator to coordinate the execution of various applications within the virtual edge instances provided to their respective tenants. For example, the edge nodes 5122, 5124 may be coordinated based on an edge provisioning function 5150, and the operation of the various applications may be coordinated with an orchestration function 5160.

[0426] The various system configurations provide an architecture that treats VMs, containers and functions equally when it comes to composing an application (and the resulting application is a combination of these three elements). Each component can use one or more accelerator (e.g. FPGA, ASIC, etc.) components as local backends. In this way, an application can be split across multiple edge owners coordinated by an orchestrator.

[0427] Additionally, in some embodiments, the compute-in-storage capabilities described in this disclosure can be leveraged throughout the edge computing system 5100 to reduce delays associated with data storage, retrieval, and computation.

[0428] It should be appreciated that the edge computing systems and arrangements described herein may be applied to a variety of solutions, services, and / or use cases. As an example, FIG. 52 illustrates a simplified vehicular computing and communication use case involving mobile access to applications in an edge computing system 5200 implementing an edge cloud 5010. In this use case, each client computing node 5210 may be implemented as an in-vehicle computing system (e.g., an in-vehicle navigation and / or infotainment system) located in a corresponding vehicle that communicates with an edge gateway node 5220 during road traversal. For example, the edge gateway node 5220 may be located in a roadside cabinet located along the road, at a road intersection, or other location near the road. As each vehicle travels along the road, the connection between its client computing node 5210 and the particular edge gateway node 5220 may be propagated to maintain a consistent connection and context for the client computing node 5210. Each of the edge gateway nodes 5220 includes some processing and storage capabilities, and as such, some processing and / or storage of data for the client computing nodes 5210 may be performed in one or more of the edge gateway nodes 5220.

[0429] Each of the edge gateway nodes 5220 may communicate with one or more edge resource nodes 5240, illustratively implemented as a computing server, appliance, or component located at or in a communications base station 5242 (e.g., a base station of a cellular network). As described above, each edge resource node 5240 includes some processing and storage capabilities, and as such, some processing and / or storage of data for the client computing nodes 5210 may be performed on the edge resource node 5240. For example, processing of less urgent or important data may be performed by the edge resource node 5240, while processing of more urgent or important data may be performed by the edge gateway device or the client node itself (e.g., depending on the capabilities of each component). Additionally, there may be various wired or wireless communication links (e.g., fiber optic wired backhaul, 5G wireless links) between the edge node 5220, the edge resource node(s) 5240, the core data center 5250, and the network cloud 5260.

[0430] The edge resource node(s) 5240 also communicate with a core data center 5250, which may include computing servers, appliances, and / or other components located at a central location (e.g., a central office of a cellular network). The core data center 5250 may provide a gateway to a global network cloud 5260 (e.g., the Internet) for the operation of the edge cloud 5010 formed by the edge resource node(s) 5240 and the edge gateway node 5220. Additionally, in some examples, the core data center 5250 may include a quantity of processing and storage capabilities, and as such, some processing and / or storage of data for client computing devices may be performed at the core data center 5250 (e.g., less urgent or critical processing, or high complexity processing). The edge gateway node 5220 or edge resource node 5240 may provide for the use of stateful applications 5232 and geographically distributed data storage 5234 (e.g., databases, data stores, etc.).

[0431] In a further example, FIG. 52 may utilize various types of mobile edge nodes, such as edge nodes hosted on vehicles (e.g., cars, trucks, trams, trains, etc.) or other mobile units, as the edge node moves along the platform that hosts it to other geographic locations. In vehicle-to-vehicle communications, individual vehicles may also act as network edge nodes for other vehicles (e.g., performing caching, reporting, data aggregation, etc.). Thus, it will be appreciated that application components provided at various edge nodes may be distributed in a variety of configurations, including coordination between some functions or operations at individual endpoint devices or edge gateway nodes 5220, some other functions or operations at edge resource nodes 5240, and other functions or operations at a core data center 5250 or global network cloud 5260.

[0432] Additionally, in some embodiments, compute-in-storage capabilities described throughout this disclosure can be leveraged throughout the edge computing system 5200 to reduce delays associated with data storage, retrieval, and computation.

[0433] In a further configuration, the edge computing system can implement FaaS or EaaS computing functions using respective executable applications or functions. In one example, a developer writes function code (e.g., “computer code” herein) representing one or more computer functions, and the function code is uploaded to a FaaS platform, provided, for example, by an edge node or a data center. A trigger, such as, for example, a service use case or an edge processing event, initiates the execution of the function code on the FaaS platform.

[0434] In an example FaaS or EaaS deployment, containers are used to provide the environment in which function code runs. A container can be any isolated execution entity, such as a process, a Docker or Kubernetes container, or a virtual machine. Edge compute systems use a variety of devices in the data center, edge, and endpoints (including mobile) to "spin up" (e.g., activate and / or allocate) the functionality that scales on demand. Function code runs on physical infrastructure devices (e.g., edge compute nodes) and the underlying virtualized containers. Finally, in response to completion of execution, the containers are "spun down" (e.g., deactivated and / or deallocated) on the infrastructure.

[0435] Further aspects of FaaS and EaaS may enable deployment of edge functions in a service manner, including support for respective functions that support edge computing as a service. Additional features of FaaS and EaaS include a granular billing component that allows customers (e.g., computer code developers) to pay only when their code is executed, common data storage to store data for reuse across one or more functions, orchestration and management between individual functions, function execution management, parallelization and integration, management of container and function memory space, tuning of acceleration resources available to functions, distribution of functions across containers (including "warm" containers that are already deployed or running and "cold" containers that need to be deployed or configured), etc.

[0436] Some of the techniques and configurations described with reference to edge computing can be implemented within a MEC environment, such as the standards and approaches published in ETSI GS MEC-003 "Mobile Edge Computing (MEC), Framework and Reference Architecture" (e.g., V2.0.3) and related MEC or network operation implementations.

[0437] MEC is intended to support the development of mobile use cases for edge computing, where application developers and content providers dynamically configure access to computing and IT service environments at the edge of the network. MEC offers application developers and content providers cloud computing and IT service environments using devices located closer to the edge of the network (e.g., cellular networks). This environment features ultra-low latency, high bandwidth, and real-time access to wireless network information that applications can exploit. MEC technology allows operators to flexibly and rapidly deploy innovative applications and services for mobile subscribers, enterprises, and vertical segments.

[0438] MEC, like other edge computing deployments, can alleviate network congestion by running applications, data functions, discovery, etc. closer to the user (e.g., mobile device, user equipment (UE), station (STA)). Details of MEC are specified in the European Telecommunications Standards Institute (ETSI) Mobile Edge Computing Introductory Technical White Paper published on September 1, 2014, including security (e.g., both user security and application integrity) and wireless usage. A series of specifications and white papers providing further details of MEC scenarios and implementation use cases are continually being developed and published by ETSI as part of the ETSI MEC Industry Specification Group (ISG).

[0439] The MEC architecture offers application developers and content providers a cloud computing and IT services environment at the edge of the network. This environment is characterized by ultra-low latency, high bandwidth, and real-time access to wireless network information that applications can exploit. Thus, MEC technology enables flexible and rapid deployment of innovative applications and services for mobile subscribers, enterprises, and vertical segments. For example, in an automotive setting, applications such as V2X (Vehicle-to-Everything, IEEE 802.11p-based or 3GPP LTE-V2X-based) may use MEC technology to exchange data, provide data to aggregation points, or access data in databases to provide and obtain an overview of the local situation from a large number of sensors (e.g. from different vehicles and roadside units).

[0440] FIG. 53 illustrates a block diagram 5300 of an example of a multi-access edge computing (MEC) system architecture. As an example, the MEC system architecture may be defined according to a specification, standard, or other definition (e.g., according to the ETSI ISG MEC-003 specification). In this diagram, the Mp reference point denotes the functionality of the MEC platform, the Mm reference point denotes management, and Mx denotes connectivity with external entities. The services, applications, orchestrators, and other entities discussed herein may be implemented in any number of entities of the MEC system architecture depicted in FIG. 53, and communications to perform network operations may be implemented in any number of interfaces of the MEC system architecture depicted in FIG. 53.

[0441] For example, a device application 5302 running on a client user equipment device (e.g., a smartphone) may access a multi-access edge orchestrator 5310, which coordinates the system configuration or characteristics of the edge compute hosts for the accomplishment of a service or application. Additionally, a particular MEC host 5350 may run one or more MEC applications 5351, 5352, 5353 or platforms 5360 that provide MEC resources or services via a virtual edge appliance. The virtualization infrastructure manager 5340 and MEC platform manager 5330 may provide management of the use of hosts, platforms, and resources, and may also provide managed access to authentication services or verifiers (not shown). The virtualization infrastructure manager 5340 and MEC platform manager 5330 may also provide managed access to other MEC hosts (e.g., host 5370) or MEC platforms (e.g., platform 5380), which may participate in the use of authentication functions as described herein.

[0442] Additionally, in some embodiments, the MEC system architecture 5300 can leverage compute-in-storage capabilities described throughout this disclosure to reduce latency associated with storing, retrieving, and computing data. As an example, the virtualization infrastructure of the MEC host 5350 can include data storage resources implemented with compute-in-storage capabilities. In some embodiments, for example, the virtualization infrastructure can include data storage devices that leverage associated processors, controllers, and / or accelerators to perform specific computational tasks on data stored in the storage devices.

[0443] At a more general level, an edge computing system may be described to encompass any number of deployments operating on an edge cloud 5010 that provides coordination from clients and distributed computing devices. Figure 54 illustrates a further abstraction of the layers of distributed computing deployed in an edge computing environment.

[0444] FIG. 54 generally depicts an edge computing system for providing edge services and applications to multi-stakeholder entities, such as those distributed across tiers of a network, such as those distributed among one or more client computing nodes 5402, one or more edge gateway nodes 5412, one or more edge aggregation nodes 5422, one or more core data centers 5432, and a global network cloud 5442. An implementation of the edge computing system may be provided by or on behalf of a telecommunications service provider ("telco", or "TSP"), an Internet-of-Things service provider, a cloud service provider (CSP), an enterprise entity, or any other number of entities. Various forms of wired or wireless connections may be configured to establish connectivity between the nodes 5402, 5412, 5422, 5432, including interconnections between such nodes (e.g., connections between edge gateway nodes 5412 and connections between edge aggregation nodes 5422).

[0445] Each node or device of the edge computing system is located in a particular layer corresponding to layers 5410, 5420, 5430, 5440, and 5450. For example, each of the client computing nodes 5402 is located in the endpoint layer 5410, while each of the edge gateway nodes 5412 is located in the edge device layer 5420 (local level) of the edge computing system. Furthermore, each of the edge aggregation nodes 5422 (and / or fog devices 5424, when deployed or operated with or between fog networking configurations 5426) is located in the network access layer 5430 (middle level). Fog computing (or fogging) generally refers to the extension of cloud computing to the edge of an enterprise network, typically in a collaborative distributed or multi-node network. Some forms of fog computing deploy compute, storage, and networking services on behalf of cloud computing locations between end devices and cloud computing data centers. Such a form of fog computation provides operations consistent with the edge computation discussed herein. Many of the aspects of edge computation discussed herein are applicable to fog networks, fogging, and fog configurations. Furthermore, aspects of the edge computation system described herein may be configured as fog, or fog aspects may be integrated into the edge computation architecture.

[0446] The core data center 5432 is located at the core network tier 5440 (e.g., a regional or geographically central level) and the global network cloud 5442 is located at the cloud data center tier 5450 (e.g., a national or global tier). Although "core" is used as a term to describe a central location (innermost part of the network) of the network that is accessible from multiple edge nodes or components, "core" does not necessarily mean the "center" or innermost part of the network. Thus, the core data center 5432 may be located in, on, or near the edge cloud 5010.

[0447] While FIG. 54 illustrates an exemplary number of client computing nodes 5402, edge gateway nodes 5412, edge aggregation nodes 5422, core data centers 5432, and global network clouds 5442, it should be understood that the edge computing system may include more or fewer devices or systems at each tier. Additionally, as illustrated in FIG. 54, the number of components at each tier 5410, 5420, 5430, 5440, 5450 generally increases at each lower level (i.e., closer to the end point). In this manner, one edge gateway node 5412 may service multiple client computing nodes 5402, and one edge aggregation node 5422 may service multiple edge gateway nodes 5412. Additionally, in some embodiments, compute-in-storage capabilities described throughout this disclosure may be leveraged throughout the edge computing system 5400 to reduce delays associated with storing, retrieving, and computing data.

[0448] Consistent with the examples provided herein, each client computing node 5402 may be implemented as any type of endpoint component, device, appliance, or "thing" capable of communicating as a producer or consumer of data. Additionally, the labels "node" or "device" used in edge computing system 5400 do not necessarily imply that such node or device operates in a client or slave role, but rather, any of the nodes or devices of edge computing system 5400 refer to an individual entity, node, or subsystem, including discrete or connected hardware or software configurations for facilitating or using the edge cloud 5010.

[0449] Thus, the edge cloud 5010 is formed from the network components and functional features operated within by the edge gateway nodes 5412 and edge aggregation nodes 5422 of layers 5420, 5430, respectively. The edge cloud 5010 may be implemented as any type of network providing edge computing and / or storage resources located in proximity to Radio Access Network (RAN) enabled endpoint devices (e.g., mobile computing devices, IoT devices, smart devices, etc.), shown in FIG. 54 as client computing nodes 5402. In other words, the edge cloud 5010 can be envisioned as an "edge" connecting endpoint devices with traditional mobile network access points, serving as an ingress point to service provider core networks including carrier networks (e.g., Global System for Mobile Communications (GSM) networks, Long Term Evolution (LTE) networks, 5G networks, etc.), and providing storage and / or computing capabilities. Other types and forms of network access (e.g., Wi-Fi, long-range wireless networks, etc.) can also be utilized instead of or in combination with such 3GPP carrier networks.

[0450] In some examples, the edge cloud 5010 may form part of or provide an ingress point to or across a fog networking configuration 5426 (e.g., a network of fog devices 5424, not shown in detail), which may be implemented as a system-level horizontal and distributed architecture that distributes resources and services to perform specific functions. For example, a coordinated distributed network of fog devices 5424 may perform computation, storage, control, or networking aspects of the IoT system configuration. Other networked, aggregated, and distributed functions may exist in the edge cloud 5010 between the core network layer 5440 and client endpoints (e.g., client compute nodes 5402). Some of these are described in the following sections in the context of virtualization of network functions and services, including the use of virtual edges and virtual services organized for multiple stakeholders.

[0451] The edge gateway nodes 5412 and edge aggregation nodes 5422 cooperate to provide various edge services and security to the client computing nodes 5402. Additionally, because each client computing node 5402 may be stationary or mobile, each edge gateway node 5412 may cooperate with other edge gateway devices to propagate the currently provided edge services and security to the corresponding client computing node 5402 as it moves within the domain. To do this, each of the edge gateway nodes 5412 and / or edge aggregation nodes 5422 may support a multi-tenant and multi-stakeholder configuration in which services from (or hosted for) multiple service providers and multiple consumers may be supported and coordinated across a single or multiple computing devices.

[0452] Within the edge cloud 5010 architecture, various security approaches can be utilized. In a multi-stakeholder environment, multiple LSMs (Loadable Security Modules) may be used to provide policies that enforce stakeholder interests. An enforcement point environment can support multiple LSMs that apply a combination of loaded LSM policies (e.g., if any stakeholder A, B, C restricts access, then the most restrictive effective policy is applied, such as restricted access). Within the edge cloud 5010, each edge entity can be provisioned with an LSM that enforces the edge entity's interests. Cloud entities can be provisioned with an LSM that enforces the cloud entity's interests. Similarly, various fog and IoT network entities can provide LSMs that enforce the fog entity's interests.

[0453] In these examples, a service can be considered in terms of transactions performed against a contract or set of components, whether at a component level or at a human perceivable level. Thus, a user who has signed a service contract with a service provider expects the service to be provided based on the terms of the SLA. Although not described in detail, the use of edge computation techniques described here may play a role in negotiating the contract and measuring the fulfillment of the contract (e.g., identifying what elements the system requires to deliver the service, and the system's response to terms and changes in the service).

[0454] "Services" is a broad term that is often applied in a variety of contexts, but generally refers to a relationship between two entities where one entity provides or performs a function for the benefit of the other. However, services provided by one entity to another must be performed under certain guidelines, ensuring a trusting relationship between the entities and governing the transaction according to contractual terms and conditions established at the beginning, middle and end of the service.

[0455] [Global service page table example] One of the challenges in a distributed architecture where data is posted across multiple edge locations for a particular service or tenants that own different services is how to manage the rights that these sets of services have to a particular set of data. Such sets of data are sometimes stored in what is known as a data lake. As an example, new hardware methods use the concept of page tables to enable efficient, low-latency data distribution (in memory) and access to devices.

[0456] Existing solutions are based on software stacks for distributed storage, which translate access from services to virtual address spaces, physical locations, and check the permissions that those services have for that particular data set. Scalability is a major limitation when using software stacks. Similar issues arise with services for tenants or devices.

[0457] The edge computing systems described herein (e.g., edge cloud 5010, and edge computing system configurations described and depicted throughout this disclosure) are extended with the implementation of edge page tables. Edge page tables are used to track which services are accessing which data, how virtual addresses from the services are mapped to physical addresses, and what the permissions on the addresses and data are. As an example, page tables allow services to be instantiated or migrated across multiple edge locations. Edge page tables may improve security and tenant isolation for endpoint devices (including protecting different regions of the page table using tenant, service, or user-based data encryption). Additionally, edge page tables may be provided on a "global" or "system-wide / multi-system" basis to improve performance of tenant-specific resource context switching.

[0458] FIG. 55 illustrates a system 5500 for implementing an edge service page table. In some examples, the edge service page table may be used with the edge cloud 5010 to manage memory pages 5520 across different edge nodes and services (including, for example, device 5530) of the edge cloud 5010. Such memory pages may be of various sizes or sets of sizes (e.g., 1 GB). For example, the computing device 5800 or edge computing device 5850 may store data, services, or other resources in the edge cloud 5010 for use by the edge devices, and the data, services, or other resources may be managed using a system-wide ("global") service page table. The global service page table may be stored in the computing device 5800 or edge computing device 5850, and elsewhere in the edge cloud 5010 (or in a computing resource such as an edge resource node 5240, such as implemented in a distributed database accessible to the core data center 5250). In one example, the global service page table may be stored on a device in the core data center 5250 or on a device in the edge cloud 5010 (e.g., a server in a cloud data center).

[0459] Page tables are used at the platform and / or CPU level to identify the rights of an application to access a particular memory page. In an edge context, page tables (or similar structures) can be used to track which edge resources or services a particular user or tenant has access to. Each edge location may have a tree-based structure with the following hierarchy: root of tree → list of services → list of providers → list of tenants or devices authorized to run the service. In a further example, the list of providers can include configuration options for hardening of workloads available to the provider. For example, VM, secure (e.g., SGX, etc.) enclave, VM+enclave, container, container+enclave, VMM, VMM+enclave, OS, FPGA, FPGA+AFU, etc. Additionally, workload combinations that include technologies such as Intel® Software Guard Extensions (SGX) (e.g., other trusted execution or trusted computing technologies applicable to x86, ARM, or other architectures) can be checked and enforced using the hardened security environment to check for expected combinations. Various adjustments or modifications of trusted execution, CPU, or platform hardware can be made to enable address space management.

[0460] In one example, the tree for tracking pages may be a virtual tree managed by a hardware entity (e.g., FPGA). If a tenant wants to execute service A from a particular edge service provider, the tenant can navigate the tree hierarchy until it reaches a leaf that contains the permissions where the particular tenant's resources, devices, storage, compute units, or communication / network units are located. In one example, there is a service that is a front-end to the tenant resources and may contain the resources (e.g., may contain metadata types such as available SLAs). If a TLB (translation look-side buffer) lookup miss occurs at any of the three layers, a page walk (similar to the process of PT) is performed and may target increasingly centralized repositories (such as a central office or core of the network) to obtain the specific permissions for that service for that user.

[0461] As an example, resource allocation may also take place. Take the page table for example: there are three things to consider in having a page table: (1) distinguishing between legal and illegal addresses (e.g., validity faults vs. segmentation faults), (2) access control (e.g., R / W and other bits like PKEY), and (3) materializing or creating resources that are available when needed (e.g., handling faults). All three are analogous to the resource management side. For example, (1) a service may be denied certain resources and allowed potential access to other resources, (2) a service may be given various percentages of resources that it may actually have access to, and the navigation of this multi-level walk may be the control point, or (3) a service may only be given resources when it actually needs them.

[0462] This approach differs from that of a centralized directory (such as LDAP), in that tenant access privileges are cached hierarchically, following a scheme that places a root cache in the core data center, with multiple copies in base stations, on-premise edge nodes, and points in between.

[0463] Tenant SLA / SLO and orchestration context can be used to proactively warm the cache based on expected location, collaboration or workload. Frequent cache access keeps cached content "warm" for the next request.

[0464] An endpoint device, host, or service may participate in a caching scheme using local storage resources. For example, local access requests do not need to leave the local platform (which may involve network latency). In one example, the access context is cached within a set of resources assigned to a particular tenant. For example, if the platform supports memory encryption (e.g., Intel® MKTME, etc.), an encrypted user key resource may be used to contain keys that are not cached by the TLB cache. Other tenant-specific resource isolation schemes may be applied on a per-platform or per-service basis. For example, persistent storage, function acceleration, CPU allocation, memory partitioning, virtualization, peripheral affinity, or IoT device network partitioning may have tenant-specific partitioning capabilities.

[0465] Changes in access policies, such as revoking a key or adding permissions to additional resources, can result in stale cached context, potentially resulting in privilege escalation or denial of service. The system 5500 uses a publish-subscribe or ICN (information-centric network) configuration to allow resource updates to be applied uniformly to all cached copies simultaneously, based on policies where caches become subscribers to tenant-specific context topics. For example, a warm cache subscribes with high QoS requirements to ensure timely updates for locally originated resource access requests.

[0466] When new data or new devices are added to the edge data lake, certain access restrictions to edge services may be applied. This information may be multicast to edges participating in the same data lake data or devices. Changes to existing data or devices may be multicast to other edges that share that data or device. As an example, a particular edge location (e.g., an at-X location in system 5500) may be assigned to data or functionality hosted within a particular edge device. The edge TLB 5510 may check whether the corresponding service has the right to access the device or data or location within the edge device.

[0467] Additionally, in some embodiments, compute-in-storage capabilities described throughout this disclosure can be utilized throughout system 5500 to reduce delays associated with data storage, retrieval, and computation.

[0468] Figure 56 illustrates an example of an edge global services page table 5610. The edge global services page table 5610 illustrates a virtual address and its corresponding physical address. The physical address in the edge global services page table 5610 may include an edge device and a physical address in memory on that particular edge device.

[0469] The edge global services page table 5610 includes local and remote memory addresses. If the edge global services page table 5610 indicates that the memory address is not local (e.g., to the edge device running the service or the edge device hosting the tenant), the edge global services page table 5610 may send a read to the remote network interface to obtain remote memory access at the remote device. In response to the remote memory access being granted, the device may be given a physical address or location for write access. In a further example, if the remote node is down, a request is sent to another replica, and the system can include a replica of that service and implement resiliency (based on the data contained in the page table). Otherwise, an error occurs in the SW stack.

[0470] As an example, other attributes may be taken into account when accessing remotely, such as security requirements by the device. In this example, it must be transmitted in a secure manner in order to return to the original platform. Another example attribute may require additional data to be added to the request, such as referencing the credentials of the remote memory (e.g., if the data in the remote memory is secure). The access policy may have metadata with characteristics indicating how much bandwidth to allocate or how much latency is required. Those characteristics may be required when returning the remote data to the local node.

[0471] In a further example, a method for implementing an edge global services page table may be implemented according to the following operations (e.g., as implemented on or by a node or device 5800, 5840, or 5850). The method includes a first operation of maintaining an edge global services page table configured to map virtual addresses to physical addresses on a plurality of edge nodes. The method includes a second operation of receiving a request to access a virtual memory address maintained in the edge global services page table. As an example, the virtual memory address corresponds to an edge service. The method includes a third operation of identifying a physical address and an edge node associated with the virtual memory address. In an example, the edge node is a remote edge node from a local node that initiated the request. The method concludes with an operation of providing access to a resource corresponding to the physical address on the edge node. As an example, the resource may include data stored at the physical address on the edge node, a service operating at the physical address on the edge node, or the location of the physical address on the edge node.

[0472] A first embodiment (embodiment F1) for implementing an edge services page table includes a device comprising a processing circuit and a memory including instructions embodied thereon that, when executed by the processing circuit, configure the processing circuit to perform operations for managing resources on an edge device including: maintaining an edge global services page table configured to map virtual addresses to physical addresses on a plurality of edge nodes, receiving a request to access a resource at a virtual memory address maintained in the edge global services page table, identifying a physical address and corresponding edge node associated with the virtual memory address, and providing access to the resource stored at the physical address.

[0473] In a second embodiment (embodiment F2), the subject matter of embodiment F1 includes providing access to a resource by sending information including a physical address and a corresponding edge node to a requesting device or service.

[0474] In a third embodiment (embodiment F3), the subject of embodiments F1-F2 includes providing access to a resource by facilitating a copy or transfer of the resource from a corresponding edge node to a requesting edge node.

[0475] In a fourth embodiment (embodiment F4), the subject of embodiments F1-F3 configures the processing circuitry to perform operations including, prior to providing access to the resource, determining whether the requesting device has authority to access the resource by checking an edge global services page table.

[0476] In a fifth embodiment (embodiment F5), subject to embodiments F1-F4, the device is an edge node of a plurality of edge nodes, and the edge global service page table is stored locally on the edge node.

[0477] In a sixth example (example F6), the subject matter of examples F1-F5 includes that providing access to the resource includes sending a read to a remote network interface to obtain remote memory access at a corresponding edge node.

[0478] In a seventh embodiment (embodiment F7), the subject of embodiments F1-F6 is that providing access to a resource includes considering security requirements of the resource, a corresponding edge node, or a requesting device.

[0479] In various environments, the embodiments F1-F7 (and other aspects of the global page table) may be observed or monitored as a result of the use of defined application programming interfaces, or interface specifications, protocols or definitions for invoking or modifying page tables, and other uses and implementations of page tables across multiple devices in an edge computing environment. The embodiments F1-F7 and other aspects of this global page configuration may also be observed or implemented as a result of service operations and service functions (e.g., for sharing pages and data structures in a FaaS or EaaS setting). Additionally, the methods of the embodiments F1-F7 may be provided to an edge computing system as implemented instructions (e.g., comprising a machine-readable medium that, when executed, performs the method) or as implemented hardware configurations (e.g., comprising an apparatus including a configuration that performs or achieves the method). Thus, the features of the embodiments F1-F7 (and other features relating to the configuration and use of the global page) may be combined with any of the other embodiments herein to configure an edge cluster or edge cloud system as coordinated or designed by a system orchestrator or architect.

[0480] [Data Aggregation Policy Example] Edge cloud infrastructure may be used to store data from multiple sources. These data may be collected at edge devices and combined in any way. Sharing data resources at edge locations may raise privacy concerns for tenants and users. The following example systems and methods address these and other issues in the data supply chain of edge computing systems (e.g., edge cloud 5010, and edge computing system configurations described and depicted throughout this disclosure), particularly privacy concerns, by enabling data aggregation. Data aggregation and processing at edge devices is useful for conserving power for low-power devices (e.g., IoT devices, etc.) and providing computation to sensors that lack or lack computing capabilities.

[0481] The following example is relevant to aggregating many types of data on edge devices: Devices such as IoT devices, cars, drones, mobile phones, and sensors often generate data for a specific purpose. Such data can be useful beyond its intended purpose, but collecting, maintaining, and evaluating the data on the device can be challenging.

[0482] For example, a mobile device (e.g., an autonomous vehicle, a drone, a mobile phone, etc.) may collect a lot of sensor data but have limited computing power to process the sensor data. The mobile device may collect data for a specific task (e.g., to keep the car on the road). The data may then become useless to the mobile device, such as after the task is accomplished (e.g., when the car passes an intersection, it may delete the sensor data related to that intersection). Edge processing nodes (e.g., base stations, small cells, etc.) may have more computing power and receive data from multiple devices, including these mobile devices.

[0483] An edge device can combine sensor data from multiple devices. For example, an edge device may receive data from multiple devices of a similar type (e.g., cars) and aggregate the data. In another example, different devices may provide data that is aggregated at the edge device. For example, an edge device can use the aggregated data to combine video footage from multiple cameras (e.g., mounted on different cars or drones) to create a 3D model of an area. Aggregating data can potentially avoid issues with individual device data (e.g., one car observing wheel slippage may result in measurement errors). With aggregated data, for example, multiple cars observing slippage in the same location may indicate that roads are becoming slippery in that location.

[0484] FIG. 57 is a diagram of a system 5700 including an architecture for aggregating data at an edge device, according to some examples.

[0485] The system 5700 includes different elements of an edge architecture that are used to aggregate data from data provider devices 5730. The system 5700 includes a platform 5710 with instructions (e.g., implementations of algorithms, logic, etc.) responsible for storing and registering aggregation functions, for example in a data manager component 5720. A registration interface allows for the registration of transformation functions. A transformation function may include the type of device 5741 or sensor to which the function is applied (e.g., a type of sensor may include the temperature measured in a car).

[0486] It should be noted that the system 5700 may include an aggregation function itself 5742. For example, given a list of data sets from edge devices, the aggregation function 5742 may perform a calculation to generate a single value to store or transmit to the cloud data provider 5750.

[0487] The system 5700 may include an aggregation interval 5743, which indicates the number of units of time available for the aggregator to store data for that device type ID before performing a calculation (e.g., a time interval of one second, one minute, etc.). In another example, the aggregation interval may include a number of receipts of data instead of or in addition to the time units.

[0488] The system 5700 includes an interface through which the device 5730 sends data to the aggregator. The aggregation logic may be responsible for implementing the interface (e.g., the interface through which the device sends data to the aggregator, or the interface with the cloud data provider). The aggregation function may be a local storage memory enforcement point where the local storage memory 5750 includes or provides a security policy to protect / obfuscate privacy-sensitive content using aggregation. (In this way, the local storage memory 5750 may apply aggregation for privacy protection even if the user has not requested aggregation.) Additionally, in some embodiments, the local storage memory 5750 may leverage compute-in-storage capabilities described throughout this disclosure to reduce delays associated with storing, retrieving, and computing data.

[0489] A computational context (e.g., context such as the type of sensor, historical trend data from the same or comparable sources, comparative values ​​of the data source, or other values ​​that indicate trust) may be used to determine the reliability of the received data. For example, data provided by a sensor that is "close" to the data being sensed may be more reliable than data from a sensor that is relatively less "close". "Close" in this context may be a physical characteristic or a connectivity characteristic (e.g., whether the sensor is validated against the system that generates the data being measured by the sensor). Also, "closeness" to the function that interprets the data may be a reliability consideration. Relatively "close" locality means that data arriving from a location that is "far" locality is more reliable. In a further example, the concept of closeness may be expanded such that distance depends on the type of provider (e.g., providers associated with a first organization may calculate distance on a first scale, such as 1 mile = 1 mile, and providers associated with a second organization may calculate distance on a second scale, such as 1 mile = 2 miles).

[0490] Edge devices can evaluate data locality as part of their trust tests. If they hold the data but communicate it back to a remote node, they may treat it as less trustworthy. If an aggregation results from a local computation (e.g. in the case of telemetry), the node may consider it to be worth higher consistency. In this example, the peer node may lower that value based on its own locality test and trust semantics (e.g. the trust of the first node).

[0491] In a further example, a method for aggregating data at an edge device may include the following operations (e.g., implemented by systems and devices in the edge cloud 5010, such as implemented on or by nodes or devices 5800, 5840, or 5850):

[0492] The method includes a first operation of receiving, at the edge node, data from a plurality of mobile devices.

[0493] The method includes a second operation of aggregating the received data using aggregation logic of the edge node. As an example, the data may be aggregated over an interval, such as a time interval, until a specified amount of data has been received (e.g., number of transmissions or packets) or until data from a specified number of devices has been received.

[0494] The method includes a third operation, at the edge node, of generating a value based on the aggregated data and the aggregation function.

[0495] The method includes a fourth operation of transmitting a value to the cloud device. As an example, the value can later be used to validate received data. In another example, the value can be used to generate a detailed map of an area. In yet another example, the value can be used to validate capabilities of the mobile device. In yet another example, the value can be used to generate usage data, provide feedback, or configure security settings.

[0496] The method includes a fifth operation of determining a confidence level of the received data. The confidence may be output together with the value. In one example, aggregating the data may include considering the confidence of the received data, e.g., determining the value based on the confidence.

[0497] A further exemplary method (Example H1) for implementing and using a multi-edge data aggregation policy is a method performed using a processing circuit (e.g., of a node or device 5800, 5840, or 5850) that includes receiving data at an edge node from multiple mobile devices, aggregating the received data using aggregation logic of the edge node, generating a value based on the aggregated data and an aggregation function at the edge node, and transmitting the value to a cloud device.

[0498] In a second embodiment (embodiment H2), the subject matter of embodiment H1 includes aggregating the received data by aggregating the received data over an interval.

[0499] In a third embodiment (embodiment H3), the subject matter of embodiment H2 includes a configuration in which the interval includes a time interval, a specified amount of data to receive, or a specified number of devices to receive the data.

[0500] In a fourth embodiment (embodiment H4), the subject of embodiments H1 to H3 includes a configuration in which the value is subsequently used to verify received data.

[0501] In a fifth example (example H5), the subject of examples H1-H4 includes a configuration in which the values ​​are used to create a detailed map of the area.

[0502] In a sixth example (example H6), the subject of examples H1-H5 includes a configuration in which the value is used to ascertain a capability of the mobile device.

[0503] In a seventh embodiment (embodiment H7), the subject of embodiments H1 to H6 includes configurations in which the value is used in generating usage data, providing feedback, or in security settings.

[0504] In an eighth embodiment (embodiment H8), the subject of embodiments H1 to H7 includes determining a reliability level of the received data.

[0505] In a ninth embodiment (embodiment H9), the subject of embodiment H8 includes outputting the value by outputting a confidence level.

[0506] In a tenth embodiment (embodiment H10), the subject of embodiments H1 to H9 includes a step of aggregating received data using the reliability of the received data.

[0507] In various environments, the embodiments H1-H10 (and other aspects of the data aggregation policy) may be observed or monitored as a result of the use of defined application programming interfaces, or interface specifications, protocols or definitions for invoking, receiving, or controlling resource operations, and other uses and implementations of policies, logic, and maps (including those communicated via data transfer) within the edge computing environment. The data aggregation policies expressed in the embodiments H1-H10 may also be implemented after or as part of operations performed within a service (e.g., implemented or accomplished as part of FaaS or EaaS). Furthermore, the methods of the embodiments H1-H10 may be provided to the edge computing system as implemented instructions (e.g., comprising a machine-readable medium that, when executed, performs the method) or as implemented hardware configurations (e.g., comprising an apparatus including a configuration that performs or accomplishes the method). Thus, the features of the embodiments H1-H10 (and other features of the data aggregation) may be combined with any of the other embodiments herein to configure an edge cluster or edge cloud system that is orchestrated or designed by a system orchestrator or architect.

[0508] [Exemplary Computing Devices] In a further example, any of the computing nodes or devices discussed with reference to the present edge computing systems and environments may be implemented based on the components depicted in Figures 58A and 58B. Each edge computing node may be implemented as a type of device, appliance, computer, or other "thing" capable of communicating with other edge, networking, or endpoint components. For example, an edge computing device may be implemented as a smartphone, a mobile computing device, a smart appliance, an in-vehicle computing system (e.g., a navigation system), or other device or system capable of performing the described functions.

[0509] In the simplified example depicted in FIG. 58A, an edge computing node 5800 includes a computing engine (also referred to herein as "computing circuitry") 5802, an input / output (I / O) subsystem 5808, data storage 5810, a communications circuitry subsystem 5812, and optionally one or more peripheral devices 5814. In other examples, each computing device may include other or additional components (e.g., a display, peripheral devices, etc.) as typically found in a computer. Furthermore, in some examples, one or more of the illustrated components may be incorporated in or form part of another component.

[0510] The computational node 5800 may be implemented as any type of engine, device, or collection of devices capable of performing various computational functions. In some examples, the computational node 5800 may be implemented as a single device, such as an integrated circuit, an embedded system, a field-programmable gate array (FPGA), a system-on-a-chip (SOC), or other integrated system or device. In an illustrative example, the computational node 5800 includes or is implemented as a processor 5804 and a memory 5806. The processor 5804 may be implemented as any type of processor capable of performing the functions described herein (e.g., executing an application). For example, the processor 5804 may be implemented as a multi-core processor(s), a microcontroller, or other processor or processing / control circuitry. In some examples, the processor 5804 may be implemented as, include, or be coupled to an FPGA, an application-specific integrated circuit (ASIC), reconfigurable hardware or hardware circuitry, or other specialized hardware to facilitate the performance of the functions described herein.

[0511] The main memory 5806 may be implemented as any type of volatile (such as, for example, DRAM (Dynamic Random Access Memory)) or non-volatile memory or data storage capable of performing the functions described herein. Volatile memory may be a storage medium that requires power to maintain the state of the data it stores. Non-limiting examples of volatile memory include various types of random access memory (RAM), such as DRAM and SRAM (Static Random Access Memory). A particular type of DRAM that may be used in the memory module is Synchronous Dynamic Random Access Memory (SDRAM).

[0512] In some examples, the memory device is a block addressable memory device, such as one based on NAND or NOR technology. The memory device may also include a three-dimensional cross-point memory device (e.g., Intel® 3D XPoint® memory, etc.) or other byte addressable write-in-place non-volatile memory device. The memory device may refer to the die itself and / or the packaged memory product. In some examples, the 3D cross-point memory (e.g., Intel® 3D XPoint® memory, etc.) may be configured with a transistorless stackable cross-point architecture in which memory cells are located at the intersections of word lines and bit lines, are individually addressable, and perform bit storage based on changes in bulk resistance. In some examples, all or a portion of the main memory 5806 may be integrated into the processor 5804. The main memory 5806 may store various software and data used during operation, such as one or more applications, data operated by the applications, libraries, and drivers.

[0513] The computation circuit 5802 is communicatively coupled to other components of the computation node 5800 via an I / O subsystem 5808, which may be implemented as circuits and / or components that facilitate input / output operations with the computation circuit 5802 (e.g., with the processor 5804 and / or main memory 5806) and other components of the computation circuit 5802. For example, the I / O subsystem 5808 may be implemented as or include a memory controller hub, an input / output control hub, an integrated sensor hub, a firmware device, a communication link (e.g., point-to-point link, bus link, wire, cable, light guide, printed circuit board trace, etc.), and / or other components and subsystems for facilitating input / output operations. In some examples, the I / O subsystem 5808 may form part of a system on a chip (SoC) and be incorporated in the computation circuit 5802 along with one or more of the processor 5804, the main memory 5806, and the other components of the computation circuit 5802.

[0514] The one or more exemplary data storage devices 5810 may be implemented as any type of device configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. Each data storage device 5810 may include a system partition that stores data and firmware code for the data storage device 5810. Each data storage device 5810 may also include one or more operating system partitions that store data files and executable files for an operating system, for example, depending on the type of computing node 5800. Additionally, in some embodiments, the data storage device 5810 may leverage compute-in-storage capabilities described throughout this disclosure to reduce delays associated with storing, retrieving, and computing data.

[0515] The communications circuitry 5812 may be embodied as any communications circuit, device, or collection thereof capable of enabling communications over a network between the computing circuitry 5802 and another computing device (e.g., edge gateway node 5412 of edge computing system 5400). The communications circuitry 5812 may be configured to effectuate such communications using any one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., cellular networking protocols such as 3GPP 4G or 5G standards, wireless local area network protocols such as IEEE 802.11 / Wi-Fi®, wireless wide area network protocols, IoT protocols such as Ethernet, Bluetooth®, Bluetooth Low Energy, IEEE 802.15.4 or ZigBee®, low power wide area network (LPWAN) or low power wide area (LPWA) protocols, etc.).

[0516] The exemplary communications circuitry 5812 includes a network interface controller (NIC) 5820, which may also be referred to as a host fabric interface (HFI). The NIC 5820 may be implemented as one or more add-in boards, daughter cards, network interface cards, controller chips, chipsets, or other devices that the compute node 5800 may use to connect with another computing device (e.g., the edge gateway node 5412). In some examples, the NIC 5820 may be implemented as part of a system-on-chip (SoC) that includes one or more processors, or may be included in a multi-chip package that also includes one or more processors. In some examples, the NIC 5820 may include a local processor (not shown) and / or local memory (not shown) that are both local to the NIC 5820. In such examples, the local processor of the NIC 5820 may be capable of performing one or more of the functions of the compute circuitry 5802 described herein. Additionally or alternatively, in such examples, the local memory of the NIC 5820 may be integrated into one or more components of the client computing node at the board level, socket level, chip level, and / or other levels.

[0517] Additionally, in some examples, each computing node 5800 may include one or more peripherals 5814. Such peripherals 5814 may include any type of peripheral found on a computing device or server, such as audio input devices, displays, other input / output devices, interface devices, and / or other peripherals, depending on the particular type of computing node 5800. In further examples, computing node 5800 may be implemented by a respective edge computing node (e.g., client computing node 5402, edge gateway node 5412, edge aggregation node 5422) in an edge computing system, or in the form of an appliance, computer, subsystem, circuit, or other component.

[0518] In a more detailed example, FIG. 58B illustrates a block diagram of an example of components that may be present in an edge computing node 5850 for implementing the techniques (e.g., operations, processes, methods, and methodologies) implemented herein. This edge computing node 5850 provides a closer view of each of the components of the node 5800 when implemented as or as part of a computing device (e.g., a mobile device, a base station, a server, a gateway, etc.). The edge computing node 5850 may include any combination of hardware or logical components mentioned herein and may include or be coupled with any device available in an edge communications network or combination of such networks. The components may be implemented as an IC adapted to the edge computing node 5850, a part thereof, a discrete electronic device, or other module, an instruction set, a programmable logic or algorithm, hardware, hardware accelerator, software, firmware, or a combination thereof, or may be an integrated component within the chassis of a larger system.

[0519] The edge computing device 5850 may include processing circuitry in the form of a processor 5852, which may be a microprocessor, a multi-core processor, a multi-threaded processor, an ultra-low voltage processor, an embedded processor, or other known processing element. The processor 5852 may be part of a system on a chip (SoC) in which the processor 5852 and other components are formed on a single integrated circuit, or in a single package, such as the Edison® or Galileo® SoC boards from Intel Corporation, Santa Clara, Calif. By way of example, the processor 5852 may include an Intel® Architecture Core® based processor, such as a Quark®, Atom®, i3, i5, i7, or MCU class processor, or another such processor available from Intel®. However, any number of other processors may be used, such as those from Advanced Micro Devices, Inc. (AMD®) of Sunnyvale, Calif., MIPS®-based designs from MIPS Technologies, Inc., ARM®-based designs licensed from ARM Holdings, Inc., or their customers or their licensees or adopters. Processors include units such as the A5-A12 processors from Apple®, the Snapdragon® processors from Qualcomm® Technologies, and the OMAP® processors from Texas Instruments.

[0520] The processor 5852 can communicate with the system memory 5854 via an interconnect 5856 (e.g., a bus). Any number of memory devices can be used to provide a particular amount of system memory. By way of example, the memory can be random access memory (RAM) conforming to JEDEC (Joint Conference on Electronic Devices Engineering) designs, such as DDR and mobile DDR standards (e.g., LPDDR, LPDDR2, LPDDR3, LPDDR4, etc.). In a particular example, the memory components can conform to DRAM standards established by JEDEC, such as JESD79F for DDR SDRAM, JESD79-2F for DDR2 SDRAM, JESD79-3F for DDR3 SDRAM, JESD79-4A for DDR4 SDRAM, JESD209 for LPDDR (Low Power DDR), JESD209-2 for LPDDR2, JESD209-3 for LPDDR3, and JESD209-4 for LPDDR4. Such standards (and similar standards) may be referred to as DDR-based standards, and the communication interface of a storage device implementing such standards may be referred to as a DDR-based interface. In various implementations, the individual memory devices may be in any number of different package types, such as single die packages (SDPs), dual die packages (DDPs), quad die packages (Q17Ps), etc. These devices may in some implementations be soldered directly onto a motherboard, providing a low profile solution, while in other implementations the devices are configured as one or more memory modules and coupled to the motherboard by a specific connector. Any number of other memory implementations may be used, such as other types of memory modules, for example, different types of dual in-line memory modules (DIMMs), including, but not limited to, microDIMMs or MiniDIMMs.

[0521] Storage 5858 may also be coupled to the processor 5852 via interconnect 5856 to provide persistent storage of information such as data, applications, and the operating system. In one example, storage 5858 may be implemented via a solid-state disk drive (SSDD). Other devices that may be used for storage 5858 include flash memory cards such as SD cards, microSD cards, XD picture cards, and USB flash drives. In one example, the memory device may be or include chalcogenide glass based memory devices, multi-threshold level NAND flash memory, NOR flash memory, single-level or multi-level phase change memory (PCM), resistive memory, nanowire memory, ferroelectric transistor random access memory (FeTRAM), antiferroelectric memory, magnetoresistive random access memory (MRAM) memory incorporating memristor technology, metal oxide based, oxygen vacancy based and conductive bridge random access memory (CB-RAM), or spin-transfer torque (STT)-MRAM, resistive memory including spintronic magnetic junction memory based devices, magnetic tunnel junction (MTJ) based devices, DW (domain wall) and SOT (spin orbit transformation) based devices, thyristor based memory devices, or devices combining any of the above, or other memories.

[0522] In low power implementations, storage 5858 may be on-die memory or registers associated with processor 5852. However, in some examples, storage 5858 may be implemented using a micro hard disk drive (HDD). Furthermore, storage 5858 may use any number of emerging technologies in addition to or in place of the technologies described, such as resistive memory, phase change memory, holographic memory, chemical memory, among others.

[0523] Additionally, in some embodiments, storage 5858 may leverage compute-in-storage capabilities described throughout this disclosure to reduce delays associated with storing, retrieving, and computing data. For example, storage 5858 may leverage associated controllers, processors (e.g., 5852), and / or accelerators (e.g., acceleration circuitry 5864) to perform certain computational tasks on data stored in storage 5858.

[0524] These components may communicate via an interconnect 5856. The interconnect 5856 may include any number of other technologies, such as ISA (Industry Standard Architecture), EISA (Extended ISA), PCI (Peripheral Component Interconnect), PCIx (Peripheral Component Interconnect Extended), PCI express (PCIe), etc. The interconnect 5856 may be a proprietary bus used in SoC-based systems, for example. Other bus systems may also be included, such as an I2C interface, an SPI interface, a point-to-point interface, a power bus, etc., among others.

[0525] The interconnect 5856 may couple the processor 5852 to a transceiver 5866 for communication with connected edge devices 5862. In particular, the transceiver 5866 may use any number of frequencies and protocols, such as the 2.4 gigahertz (GHz) transmission of the IEEE 802.15.4 standard, using the Bluetooth low energy (BLE) standard defined by the Bluetooth Special Interest Group, or using the ZigBee standard. Any number of radios configured for a particular wireless communication protocol may be used to connect to the connected edge devices 5862. For example, a WLAN (Wireless Local Area Network) unit may be used to achieve Wi-Fi (Wireless Local Area Network) communication in accordance with the IEEE (Institute of Electrical and Electronics Engineers) 802.11 standard. Furthermore, wireless wide area communication may be performed via a wireless wide area network (WWAN) unit, for example according to a cellular or other wireless wide area protocol.

[0526] The wireless network transceiver 5866 (or multiple transceivers) may communicate using multiple standards or radios for communication at different ranges. For example, the edge computing node 5850 may communicate with nearby devices, e.g., within about 10 meters, using a local transceiver based on BLE, or other low power radio, to conserve power. Connected edge devices 5862 that are further away (e.g., within about 50 meters) can be reached via ZigBee or other medium power radio. Both communication technologies may be on a single radio at different power levels, or on separate transceivers, e.g., a local transceiver using BLE and another mesh transceiver using ZigBee.

[0527] A wireless network transceiver 5866 (e.g., a radio transceiver) may be included to communicate with devices or services in the edge cloud 5890 via local or wide area network protocols. The wireless network transceiver 5866 may be an LPWA transceiver conforming to the IEEE 802.15.4 or IEEE 802.15.4g standards, among others. The edge computing node 5850 may communicate over a wide area using LoRaWAN (Long Range Wide Area Network) developed by Semtech and the LoRa Alliance. The techniques described herein are not limited to these techniques and may be used with any number of other cloud transceivers implementing long range, low bandwidth communication, such as Sigfox. Additionally, other communication techniques may be used, such as time slotted channel hopping as described in the IEEE 802.15.4e specification.

[0528] In addition to the systems mentioned for the wireless network transceiver 5866 as described herein, any number of other wireless communications and protocols may be used. For example, the transceiver 5866 may include a cellular transceiver using spread spectrum (SPA / SAS) communications to achieve high speed communications. Additionally, any number of other protocols, such as Wi-Fi® networks, may be used to provide medium speed or network communications. The transceiver 5866 may include radios compatible with any number of 3GPP (Third Generation Partnership Project) specifications, such as LTE (Long Term Evolution) and 5G (Fifth Generation) communications systems, which are described in more detail at the end of this disclosure. A network interface controller (NIC) 5868 may be included to provide wired communications to nodes of the edge cloud 5890 or other devices, such as connected edge devices 5862 (e.g., operating in a mesh). Wired communications can use Ethernet connections as well as other type-based networks such as CAN (Controller Area Network), LIN (Local Interconnect Network), DeviceNet, ControlNet, Data Highway+, PROFIBUS, PROFINET, etc. Additional NICs 5868 may be included to allow connection to a second network, for example a first NIC 5868 providing communications over Ethernet to the cloud and a second NIC 5868 providing communications to other devices over another type of network.

[0529] Because of the variety of applicable types of communication from the device to another component or network, applicable communication circuitry used by the device may include or be implemented by any one or more of components 5864, 5866, 5868, or 5870. Thus, in various examples, applicable means for communicating (e.g., receiving, transmitting, etc.) may be implemented by such communication circuitry.

[0530] The edge compute node 5850 may include or be coupled to acceleration circuitry 5864, which may be implemented by one or more AI accelerators, neural compute sticks, neuromorphic hardware, FPGAs, arrays of GPUs, one or more SoCs, one or more CPUs, one or more digital signal processors, special purpose ASICs, or other forms of specialized processors or circuits designed to accomplish one or more specialized tasks. These tasks include AI processing (including machine learning, training, inference, and classification operations), vision data processing, network data processing, object detection, rules analysis, etc.

[0531] The interconnect 5856 may couple the processor 5852 to a sensor hub or external interface 5870 used to connect additional devices or subsystems. The devices may include sensors 5872, such as accelerometers, level sensors, flow sensors, light sensors, camera sensors, temperature sensors, global navigation system (e.g., GPS) sensors, pressure sensors, barometric pressure sensors, etc. The hub or interface 5870 may further be used to connect the edge computing node 5850 to actuators 5874, such as power switches, valve actuators, audible sound generators, visual warning devices, etc.

[0532] In some example options, various input / output (I / O) devices may be present in or connected to the edge computing node 5850. For example, a display or other output device 5884 may be included to display information such as sensor readings or actuator positions. Input devices 5886, such as a touch screen or keypad, may be included to accept input. The output devices 5884 may include any number of forms of audio or visual display, including simple visual output such as binary status indicators (e.g., LEDs) or multi-character visual output, or more complex output such as a display screen (e.g., LCD screen) where output such as text, graphics, multimedia objects, etc. are generated or generated from the operation of the edge computing node 5850. The display or console hardware may be used in the context of the present system to provide edge computing system output and receive input, to manage edge computing system components or services, to identify the status of edge computing components or services, or to perform any number of other management or operational functions or service use cases.

[0533] The battery 5876 may provide power to the edge computing node 5850, but in instances where the edge computing node 5850 is mounted in a fixed location, it may have a power source tied to an electrical grid, or the battery may be used for backup or temporary functions. The battery 5876 may be a lithium-ion battery or a metal-air battery, such as a zinc-air battery, an aluminum-air battery, or a lithium-air battery.

[0534] A battery monitor / charger 5878, if included, may be included in the edge compute node 5850 to track the state of charge (SoCh) of the battery 5876. The battery monitor / charger 5878 may be used to monitor other parameters of the battery 5876 to provide failure predictions, such as the state of health (SoH) and state of function (SoF) of the battery 5876. The battery monitor / charger 5878 may include a battery monitor integrated circuit, such as the LTC4020 or LTC2990 from Linear Technologies, the ADT7488A from ON Semiconductor of Phoenix, Arizona, or the UCD90xxx family of ICs from Texas Instruments of Dallas, Texas. The battery monitor / charger 5878 may communicate information about the battery 5876 to the processor 5852 via the interconnect 5856. The battery monitor / charger 5878 may also include an analog-to-digital (ADC) converter that allows the processor 5852 to directly monitor the voltage of the battery 5876 or the flow of current from the battery 5876. The battery parameters may be used to determine the operations that the edge computing node 5850 can perform, such as transmission frequency, mesh network operation, sensing frequency, etc.

[0535] A power generation block 5880, or other power source coupled to the grid, may be coupled to a battery monitor / charger 5878 to charge the battery 5876. In some examples, the power generation block 5880 may be replaced with a wireless power receiver to obtain power wirelessly, for example, via a loop antenna of the edge compute node 5850. In particular, a wireless battery charging circuit, such as an LTC4020 chip from Linear Technologies, Inc., Milpitas, Calif., may be included in the battery monitor / charger 5878. The particular charging circuit may be selected based on the size of the battery 5876 and thus the current required. In particular, charging may be performed using the Airfuel standard developed by the Airfuel Alliance, the Qi wireless charging standard developed by the Wireless Power Consortium, the Rezence charging standard developed by the Alliance for Wireless Power, etc.

[0536] Storage 5858 may include instructions 5882 in the form of software, firmware, or hardware commands for implementing the techniques described herein. Although such instructions 5882 are shown as code blocks contained in memory 5854 and storage 5858, it should be understood that any of the code blocks may be replaced by hardwired circuitry, for example, incorporated into an ASIC (application specific integrated circuit).

[0537] In one example, the instructions 5882 provided via the memory 5854, storage 5858, or processor 5852 may be embodied as a non-transitory machine-readable medium 5860 including code that instructs the processor 5852 to perform electronic operations at the edge computing node 5850. The processor 5852 may access the non-transitory machine-readable medium 5860 via the interconnect 5856. For example, the non-transitory machine-readable medium 5860 may be embodied by a device described for the storage 5858 and may include a specific storage unit such as an optical disk, a flash drive, or any number of other hardware devices. The non-transitory machine-readable medium 5860 may include instructions that instruct the processor 5852 to perform a particular sequence or flow of operations, for example, as described with respect to the flowchart(s) and block diagram(s) of operations and functionality depicted above. As used herein, the terms "machine-readable medium" and "computer-readable medium" are interchangeable.

[0538] In a further example, a machine-readable medium includes any tangible medium that can store, encode, or carry instructions executed by a machine and cause a machine to perform any one or more of the methods of the present disclosure, or that can store, encode, or carry data structures used by or associated with such instructions. A "machine-readable medium" includes, but is not limited to, solid-state memory, optical media, magnetic media, and the like. Specific examples of machine-readable media include non-volatile memory, including, but not limited to, semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Instructions embodied by a machine-readable medium can further be transmitted or received over a communication network using a transmission medium via a network interface device utilizing any one of a number of transfer protocols (e.g., HTTP).

[0539] The machine-readable medium may be provided by a storage device or other device capable of hosting data in a non-transitory format. As an example, the information stored or otherwise provided on the machine-readable medium is representative of instructions, such as the instructions themselves or a format from which instructions can be derived. Such formats from which instructions can be derived may include source code, encoded instructions (e.g., compressed or encrypted format), packaged instructions (e.g., split into multiple packages), and the like. The information representative of instructions in the machine-readable medium may be processed by a processing circuit into instructions to perform any of the operations described herein. For example, deriving instructions from information (e.g., processing by a processing circuit) may include compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, erasure coding, encrypting, decrypting, packaging, unpackaging, or otherwise manipulating the information to convert it into instructions.

[0540] As an example, deriving instructions may include assembling, compiling, or interpreting information (e.g., by processing circuitry) to create the instructions from some intermediate or preprocessed format provided by a machine-readable medium. If the information is provided in multiple parts, they may be combined, decompressed, or modified to create the instructions. For example, the information may be multiple compressed source code packages (or object code, or binary executable code, etc.) on one or more remote servers. The source code packages may be encrypted as they travel across the network, and then decrypted, decompressed, and assembled (e.g., linked, etc.) as needed, compiled, or interpreted (e.g., converted into libraries, standalone executables, etc.) on the local machine for execution.

[0541] FIG. 58C illustrates Rack Scale Design (RSD) components that may be included as part of a server or other discrete compute node in an edge platform architecture. This arrangement provides a close view of the configurable processing components of a node 5800 or device 5850 when implemented as a server (e.g., server rack, blade, etc.). This configurable architecture differs from other architectures in its separate use of FPGA (field programmable gate array), NVMe (non-volatile memory express), and I / O (input-output) pooling resources. The FPGA and NVMe resources provide elements that can be used for any type of edge service, such as video or voice analytics. The I / O pooling is used to provide flexible network capabilities. This architecture allows network interfaces to scale depending on the expected data rate and network load for a particular virtual network function (VNF). This architecture also provides the resilience to map different network cards to compute nodes depending on the type of network processing that takes place at each node.

[0542] The illustrated RSD architecture includes a Point of Delivery (POD) manager 5842. The POD manager 5842 is responsible for managing resources, including compute and isolation resources, in a POD (e.g., one or more racks). The POD manager 5842 exposes an interface to the orchestrator to create, manage, or destroy composite nodes. Managing composite nodes includes features to scale up or down the amount of pooled resources 5848 connected to a particular compute thread 5840. The POD manager 5842 typically runs on a node controller. The POD manager 5842 discovers resources in a POD, configures and manages resources, configures logical servers, etc. As an example, the POD manager 5842 is an optional separate component and is not required in a rack. However, as an example, to be "RSD compliant," a certified POD manager is required to be able to manage a rack.

[0543] Some examples of attributes of POD manager 5842 are shown below. For example, a rack can include a set of compute threads 5840 that are used to execute edge services and other related system software stacks (e.g., orchestration or other system services, etc.). One type of compute thread 5840 can be a Pooled Resources Sled. This compute thread 5840 can manage a set of distributed resources. Here, the compute thread 5840 can include a pooling system management engine software (PSME) 5841. The PSME 5841 provides a management interface for managing modules or blades at a drawer level. As an example, a rack can include one or more logical PSMEs. For example, each drawer can have a PSME, or server drawers can share a PSME, or the PSME can run on a top-of-rack (TOR) 5844 switch or on another host. As an example, the PSME 5841 supports the RSD API.

[0544] In one example, the compute thread 5840 includes a processor (e.g., CLX) running an RSD software stack implementing an NVM-oF or FPGA-oF that acts as a target system and can manage a set of distributed resources. In one example, the processor is connected using a PCIe x16 bifurcated port to a PCIe switch 5846 that provides access to the target resources (FPGA or NVME in the RSD 5848).

[0545] Various RSD edge composite node flavors can be used by the compute threads 5840 to run edge services. Services running on these nodes can use client software libraries or drivers to provide transparent access to the distributed FPGAs and NVME in the RSD 5848. In a further example, the rack includes one or more PCIE switches that connect the compute threads 5840 to a set of distributed resources (e.g., the RSD 5848).

[0546] The block diagrams of Figures 58A, 58B, and 58C are intended to depict high-level views of components of varying devices, subsystems, or arrangements of edge computing nodes. However, it should be understood that other implementations may omit some of the illustrated components, may have additional components present, or may result in different arrangements of the illustrated components. Moreover, these arrangements may be used in a variety of use cases and environments, including those described below (e.g., mobile UEs in industrial computing for smart cities and smart factories, among many other examples).

[0547] Each of the compute platforms in Figures 58A, 58B, and 58C may support multiple edge instances (e.g., edge clusters) through the use of tenan...

Claims

1. A non-volatile storage medium; receiving a chained computation offload command from a host computing device via an interface circuit, the chained computation offload command instructing a storage device to perform a plurality of chained operations on an input data object stored in the non-volatile storage medium; reading the input data object from the non-volatile storage medium; performing the chained operations on the input data object, where an output data object is calculated based on the execution of the chained operations; returning the output data object as an output of the chained computation offload command; A processing circuit for performing A storage device comprising:

2. 2. The memory device of claim 1, wherein an output of a first operation of the chained plurality of operations becomes an input of a second operation chained to the first operation of the chained plurality of operations.

3. The chain computation offload command is an input parameter indicating a first storage location of the input data object on the non-volatile storage medium; an operation chain parameter indicating an order in which the operations are to be executed by describing the operations in an execution order; an operation argument parameter indicating arguments for the chained operations; Equipped The storage device according to claim 1 .

4. The chain computation offload command is An output parameter indicating how the output data object is returned Further equipped The storage device according to claim 3.

5. the output parameter indicates that the output data object is to be written to the non-volatile storage medium; The processing circuitry that returns the output data object as the output of the chained computation offload command, writing the output data object to the non-volatile storage medium at a second storage location; transmitting an address of the second storage location to the host computing device via the interface circuit; Run more The storage device according to claim 4.

6. The storage device of claim 5 , wherein the second storage location comprises a pre-allocated output buffer on the non-volatile storage medium.

7. the output parameter indicates that the output data object is to be written to a second location on the non-volatile storage medium; The processing circuitry that returns the output data object as the output of the chained computation offload command, further performing the step of writing the output data object to the non-volatile storage medium at the second storage location indicated by the output parameter. The storage device according to claim 4.

8. the output parameter indicates that the output data object is to be sent to the host computing device; The processing circuitry that returns the output data object as the output of the chained computation offload command, transmitting said output data object to said host computing device via said interface circuitry. Run more The storage device according to claim 4.

9. The processing circuitry for performing the chained operations on the input data object comprises: executing the chained operations based on the order indicated by the operation chaining parameter and the arguments indicated by the operation argument parameter; computing a plurality of intermediate outputs of the plurality of chained operations, each intermediate output of a particular chained operation being input to a corresponding subsequent chained operation based on the order indicated by the operation chaining parameter; and calculating the output data object based on the intermediate outputs; Run more The storage device according to any one of claims 3 to 8.

10. The processing circuitry for computing the intermediate outputs of the chained operations comprises: writing at least a subset of the plurality of intermediate outputs to a buffer pool on the non-volatile storage medium; returning an address of each intermediate output in the buffer pool to the corresponding subsequent chained operation; Run more The storage device according to claim 9.

11. if a data size of the at least a subset of the plurality of intermediate outputs is greater than a configurable threshold, the at least a subset of the plurality of intermediate outputs is written to a buffer pool on the non-volatile storage medium; 11. The storage device of claim 10, wherein if the data size of the at least a subset of the plurality of intermediate outputs is less than or equal to the configurable threshold, the at least a subset of the plurality of intermediate outputs is passed as an input to the corresponding subsequent chained operation based on the order indicated by the operation chaining parameter.

12. the non-volatile storage medium includes a non-volatile memory; The storage device is a block storage device, the block storage device providing block-based storage to the non-volatile memory. The storage device according to any one of claims 1 to 11.

13. A processing circuit of the storage device receiving a chained computation offload command from a host computing device via an interface circuit, the chained computation offload command instructing the storage device to perform a plurality of chained operations on an input data object stored in a non-volatile storage medium of the storage device; reading the input data object from the non-volatile storage medium; performing the chained operations on the input data object, where an output data object is calculated based on the execution of the chained operations; returning the output data object as an output of the chained computation offload command; A program for executing the above.

14. The chain computation offload command is an input parameter indicating a first storage location of the input data object on the non-volatile storage medium; an operation chain parameter indicating an order in which the operations are to be executed by describing the operations in an execution order; an operation argument parameter indicating arguments for the chained operations; Equipped The program according to claim 13.

15. The chained computation offload command further comprises an output parameter indicating how the output data object is returned. The program according to claim 14.

16. the output parameter indicates that the output data object is to be written to the non-volatile storage medium; returning the output data object as the output of the chained computation offload command, writing the output data object to the non-volatile storage medium at a second location; transmitting an address of the second storage location to the host computing device via the interface circuit; Includes The program according to claim 15.

17. 20. The program product of claim 16, wherein the second storage location comprises a pre-allocated output buffer on the non-volatile storage medium.

18. the output parameter indicates that the output data object is to be written to a second location on the non-volatile storage medium; The step of returning the output data object as the output of the chained computation offload command further comprises: writing the output data object to the non-volatile storage medium at the second storage location indicated by the output parameter. The program according to claim 15.

19. the output parameter indicates that the output data object is to be sent to the host computing device; returning the output data object as the output of the chained computation offload command, transmitting said output data object to said host computing device via said interface circuit. The program according to claim 15.

20. performing the chained operations on the input data object, further comprising: executing the chained operations based on the order indicated by the operation chaining parameter and the arguments indicated by the operation argument parameter; computing a plurality of intermediate outputs of the plurality of chained operations, each intermediate output of a particular chained operation being input to a corresponding subsequent chained operation based on the order indicated by the operation chaining parameter; calculating the output data object based on the intermediate outputs; Includes 20. The program according to any one of claims 14 to 19.

21. The step of computing the intermediate outputs of the chained operations further comprises: writing at least a subset of the plurality of intermediate outputs to a buffer pool on the non-volatile storage medium; returning an address of each intermediate output in the buffer pool to the corresponding subsequent chained operation; Includes The program according to claim 20.

22. A machine-readable storage medium storing a program according to any one of claims 13 to 21.

23. 1. A method performed by a processing circuit of a storage device, comprising: receiving a chained computation offload command from a host computing device, the chained computation offload command instructing the storage device to perform a plurality of chained operations on an input data object stored in a non-volatile storage medium of the storage device; reading the input data object from the non-volatile storage medium; performing the chained operations on the input data object, where an output data object is calculated based on the execution of the chained operations; returning the output data object as an output of the chained computation offload command; A method comprising:

24. The chain computation offload command is an input parameter indicating a first storage location of the input data object on the non-volatile storage medium; an operation chain parameter indicating an order in which the operations are to be executed by describing the operations in an execution order; an operation argument parameter indicating arguments for the chained operations; Equipped 24. The method of claim 23.

25. performing the chained operations on the input data object, comprising: executing the chained operations based on the order indicated by the operation chaining parameter and the arguments indicated by the operation argument parameter; computing a plurality of intermediate outputs of the chained plurality of operations, each intermediate output of a particular chained operation being input to a corresponding subsequent chained operation based on the order indicated by the operation chaining parameter; calculating the output data object based on the intermediate outputs; Includes 25. The method of claim 24.

26. A non-volatile storage medium; receiving a chained computation offload command from a host computing device, the chained computation offload command instructing a storage device to perform a plurality of chained operations on an input data object stored in the non-volatile storage medium; reading the input data object from the non-volatile storage medium; performing the chained operations on the input data object, where an output data object is calculated based on the execution of the chained operations; and returning the output data object as an output of the chained computation offload command; and the means to carry out A storage device comprising:

27. The chain computation offload command is an input parameter indicating a first storage location of the input data object on the non-volatile storage medium; an operation chain parameter indicating an order in which the operations are to be executed by describing the operations in an execution order; an operation argument parameter indicating arguments for the chained operations; Equipped 27. The storage device of claim 26.

28. The means for performing the chained operations on the input data object comprises: executing the chained operations based on the order indicated by the operation chaining parameter and the arguments indicated by the operation argument parameter; computing a plurality of intermediate outputs of the chained operations, each intermediate output of a particular chained operation being input to a corresponding subsequent chained operation based on the order indicated by the operation chaining parameter; calculating the output data object based on the intermediate outputs; have the means to carry out 28. The storage device of claim 27.

Images