Near data processing (NDP) in network nodes

Abstract

In some implementations, a storage node receives data processing instructions from a compute node of a network architecture. The storage node detects, within the data processing instructions, a directed acyclic graph (DAG). The DAG includes a plurality of data processing operations arranged according to an execution order within the DAG. The data processing operations further include user-defined function (UDF) information associated with a UDF. The storage node can retrieve, from an object store of the distributed storage node, a first data object based on object key information within the plurality of data processing operations. The storage node can execute the plurality of data processing operations using the retrieved first data object to generate a DAG execution result, the execution of the plurality of data processing operations based on the execution order. The storage node can communicate the DAG execution result to the compute node in response to the data processing instructions.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to data processing in disaggregated architectures. Some aspects relate to near data processing (NDP) in network nodes such as distributed storage nodes (or other types of network nodes) within a storage network. BACKGROUND

[0002] Public cloud networks (e.g., big data analytics (BDA) or artificial intelligence (AI) based networks) typically employ a compute and storage disaggregated architecture, i.e., an architecture in which the two components are managed and connected independently using an intermediate network. While the disaggregated architecture allows independent scaling of compute and storage, which simplifies storage management and reduces its cost, such an architecture is associated with several drawbacks. For example, the intermediate network connecting the compute layer and the storage layer can be a significant performance bottleneck due to the internal bandwidth of storage devices being much higher than the external network bandwidth. As a result, platforms running on the disaggregated architecture can not perform as well as platforms running on a traditional shared nothing architecture. SUMMARY

[0003] Various examples are now described, by way of example, with reference to the drawings, of which:

[0004] According to a first aspect of the present disclosure, there is provided a computer-implemented method for near data processing (NDP) within a distributed storage node of a network architecture. The method comprises receiving, at the distributed storage node, data processing instructions from a compute node of the network architecture. The method further comprises detecting, within the data processing instructions, a directed acyclic graph (DAG). The DAG comprises a plurality of data processing operations. The plurality of data processing operations are arranged within the DAG according to an execution order and comprise user-defined function (UDF) information associated with a UDF. The method further comprises retrieving, from an object store of the distributed storage node, a first data object based on object key information within the plurality of data processing operations. The method further comprises performing the plurality of data processing operations using the retrieved first data object to execute the UDF and generate a DAG execution result, the performance of the plurality of data processing operations being based on the execution order. The method further comprises communicating, responsive to the data processing instructions, the DAG execution result to the compute node.

[0005] In a first implementation form of the method according to the first aspect as such, the method further comprises detecting that the data processing instruction comprises the DAG and a legacy data processing operation. The method further comprises routing the legacy data processing operation to an object service of the distributed storage node for execution using a second data object from an object store. The second data object is selected based on second object key information within the legacy data processing operation.

[0006] In a second implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, wherein the legacy data processing operation comprises a non-complex query-based operation or a primitive request operation. The primitive request operation comprises at least one of: a get operation, a put operation, a paste operation, and a delete operation. The non-complex query-based operation comprises at least one of: a filter operation and an aggregate operation.

[0007] In a third implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the DAG is routed to an NDP service of the distributed storage node. The NDP service configures virtual resources of the distributed storage node for performing at least the retrieval of the first data object and the execution of the plurality of data processing operations using the retrieved first data object.

[0008] In a fourth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, wherein the plurality of data processing operations comprises a complex query-based operation. The complex query-based operation further comprises object key information and object location information specifying a storage bucket of an object store storing the first data object.

[0009] In a fifth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, wherein the UDF information comprises a UDF application code of the UDF and the object key information.

[0010] In a sixth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the UDF application code is executed using a serverless framework of the distributed storage node to perform the UDF as a runtime query-based invocation of a serverless function using the first data object associated with the object key information.

[0011] In a seventh implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, wherein performing the runtime query-based invocation of the UDF further comprises deploying a plurality of container-based function invocations within the serverless framework of the distributed storage node to execute the UDF.

[0012] In an eighth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, wherein the plurality of container-based function invocations within the serverless framework of the distributed storage node are configured to exchange state information over shared storage within the object storage of the distributed storage node.

[0013] In a ninth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the computer-implemented method further comprises executing the UDF application code using a virtual machine or a container instantiated within the distributed storage node.

[0014] In a tenth implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, the UDF information comprises a request to invoke the UDF for the first data object, the UDF information further specifying a location within a function registry of the object storage where UDF application code of the UDF is stored, the computer-implemented method further comprising executing the UDF application code according to the function registry to perform the UDF as a serverless function runtime-based query-dependent invocation.

[0015] In an eleventh implementation form of the method according to the first aspect as such or any preceding implementation form of the first aspect, wherein the UDF information specifies an object storage event as an invocation trigger for an event-driven invocation of the UDF as a serverless function, the UDF information further specifying a location within a function registry of the object storage where UDF application code of the UDF is stored, the computer-implemented method further comprising detecting the object storage event specified by the UDF information, the object storage event being associated with a data update of the object storage. Based on the detection of the object storage event, the UDF application code is executed according to the function registry to perform the event-driven invocation of the UDF as a serverless function.

[0016] According to a second aspect of the present application, there is provided a system comprising a memory storing instructions and one or more processors in communication with the memory. The one or more processors execute the instructions to receive, at a distributed storage node, a data processing instruction from a compute node of a network architecture. A directed acyclic graph (DAG) is detected within the data processing instruction. The DAG comprises a plurality of data processing operations. The plurality of data processing operations are arranged within the DAG according to an execution order and comprise user-defined function (UDF) information associated with a UDF. A first data object is retrieved from an object store of the distributed storage node based on object key information within the plurality of data processing operations. The plurality of data processing operations are executed using the retrieved first data object to execute the UDF and generate a DAG execution result. Execution of the plurality of data processing operations is based on the execution order. The DAG execution result is communicated to the compute node in response to the data processing instruction.

[0017] In a first implementation form of the system according to the second aspect as such, the UDF information comprises a UDF application code of the UDF and the object key information, and the one or more processors execute the instructions to execute the UDF application code using a serverless framework of the distributed storage node to execute the UDF as a runtime query-based invocation of a serverless function using the first data object associated with the object key information.

[0018] In a second implementation form of the system according to the second aspect as such or any preceding implementation form of the second aspect, wherein to execute the UDF as the runtime query-based invocation of the serverless function, the one or more processors execute the instructions to deploy a plurality of container-based function invocations within the serverless framework of the distributed storage node to execute the UDF.

[0019] In a third implementation form of the system according to the second aspect as such or any preceding implementation form of the second aspect, wherein the plurality of container-based function invocations within the serverless framework of the distributed storage node are configured to exchange state information through shared storage within the object store of the distributed storage node.

[0020] According to a third aspect of the present application, there is provided a non-transitory computer-readable medium storing instructions for near data processing (NDP) within a distributed storage node of a network architecture, the instructions, when executed by one or more processors, cause the one or more processors to perform operations. The operations comprise receiving, at the distributed storage node, a data processing instruction from a compute node of the network architecture. A loop-free directed acyclic graph (DAG) is detected within the data processing instruction. The DAG comprises a plurality of data processing operations. The plurality of data processing operations are arranged within the DAG according to an execution order and comprise user-defined function (UDF) information associated with a UDF. A first data object is retrieved from an object store of the distributed storage node based on object key information within the plurality of data processing operations. The plurality of data processing operations are executed using the retrieved first data object to execute the UDF and generate a DAG execution result. Execution of the plurality of data processing operations is based on the execution order. The DAG execution result is communicated to the compute node in response to the data processing instruction.

[0021] In a first implementation form of the non-transitory computer-readable medium according to the third aspect as such, wherein the UDF information comprises a UDF application code of the UDF and the object key information, the instructions further cause the one or more processors to execute the UDF application code using a virtual machine or a container instantiated within the distributed storage node.

[0022] In a second implementation form of the non-transitory computer-readable medium according to the third aspect as such, wherein the UDF information comprises a request to invoke the UDF on the first data object, the UDF information further specifies a location within a function registry of the object store storing a UDF application code of the UDF, the instructions further cause the one or more processors to perform operations comprising: executing the UDF application code according to the function registry to execute the UDF as a serverless function runtime based on a query invocation.

[0023] In a third implementation of the third aspect using a non-transitory computer-readable medium, wherein the UDF information specifies an object storage event as a call trigger for an event-driven invocation of the UDF as a serverless function, the UDF information further specifies the location within the function registry of the object storage where the UDF application code of the UDF is stored, and the instructions further cause the one or more processors to perform an operation including: detecting the object storage event specified by the UDF information, the object storage event being associated with a data update in the object storage. The operation further includes, based on the detection of the object storage event, executing the UDF application code according to the function registry to execute the event-driven invocation of the UDF as a serverless function.

[0024] Any of the foregoing examples can be combined with any one or more of the other foregoing examples to produce new embodiments within the scope of the invention. Attached Figure Description

[0025] The diagrams generally illustrate the various embodiments discussed herein by way of example rather than limitation.

[0026] Figure 1 A high-level system overview of a network architecture using near data processing (NDP) services for some exemplary embodiments.

[0027] Figure 2 A block diagram illustrating a distributed storage node with NDP service, showing some exemplary embodiments.

[0028] Figure 3 A block diagram illustrating the generation of an acyclic directed graph (DAG) for an NDP in a distributed storage node, for the purpose of demonstrating some exemplary embodiments.

[0029] Figure 4 A block diagram of an exemplary DAG with user-defined functions (UDFs) is provided to illustrate some exemplary embodiments.

[0030] Figure 5 Some exemplary embodiments are shown by Figure 2 The NDP service is used for serverless frameworks for UDF processing.

[0031] Figure 6 A block diagram illustrating the runtime of a UDF within a distributed storage node using query-based invocation and event-driven invocation, to demonstrate some exemplary embodiments.

[0032] Figure 7A flowchart illustrating an NDP method applicable to a distributed storage node within a network architecture, for some exemplary embodiments.

[0033] Figure 8 The block diagrams provided are intended to illustrate representative software architectures of some exemplary embodiments, which can be used in conjunction with various device hardware described herein.

[0034] Figure 9 A block diagram of the circuit of a device for illustrating implementation algorithms and execution methods of some exemplary embodiments. Detailed Implementation

[0035] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, any number of techniques can be used to implement the above. Figures 1 to 9 The disclosed systems and / or methods described are either currently known or not yet existing. This invention should in no way be limited to the illustrative implementations, figures, and techniques shown below, including the exemplary designs and implementations shown and described herein, but can be modified within the full scope of the appended claims and their equivalents.

[0036] The following is a detailed description in conjunction with the accompanying drawings, which are an integral part of the description and illustrate specific embodiments that can be practiced. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the subject matter of the invention. It should be understood that other embodiments can be utilized, and structural, logical, and electrical changes can be made without departing from the scope of the invention. Therefore, the following description of exemplary embodiments is not intended to be limiting, and the scope of the invention is defined by the appended claims.

[0037] As used herein, the term "network-based service infrastructure" includes multiple network devices (also referred to as hosts, nodes, or servers) that provide on-demand computing power (e.g., via one or more virtual machines or other virtual resources running on the network devices) and storage capacity as a service to a group of end-users (e.g., customers of the service infrastructure), who are coupled to the network devices within the service infrastructure via network communications. Customers of the service infrastructure can use one or more computing devices (or client devices) to access and manage services (including any Function as a Service (or FaaS)) provided by the service infrastructure over the network. Client devices, the network, and the network-based service infrastructure can be collectively referred to as the "network architecture." Customers of the service infrastructure can also be referred to as "users."

[0038] As used herein, the term "decomposed architecture" refers to a network architecture that decomposes compute and storage and uses separate compute and storage networks.

[0039] As used herein, the term “near data processing (NDP)” indicates placing processing power near the data (e.g., as part of a distributed storage node within a storage network) rather than shipping the data for processing to compute nodes within a compute network.

[0040] Different analysis / computing platforms (e.g., and Each of these has its own computational analysis mechanisms, which are typically only understood by the computing platform itself. As used in this paper, the term "semantically aware pushdown" (or "pushdown") indicates that NDPs are pushed down to storage nodes, rather than pushing down data processing across the entire computing platform. Therefore, data processing functions previously performed by computing nodes are now performed within distributed storage nodes (e.g., by publicly available NDP services).

[0041] As used herein, the term "serverless computing" or "serverless framework," also known as FaaS, refers to the execution of application logic without requiring feature developers and operators to be aware of deployment, scaling, capacity planning, and maintenance operations. Traditionally, serverless frameworks run solely on the compute side and are stateless. Using publicly available technologies, serverless frameworks are used as part of an NDP service within a distributed storage node, where the NDP needs to run serverlessly and maintain some state within the storage.

[0042] As used in this article, the term "query" for any data refers to a query in the style of a structured query language (SQL) for structured, semi-structured, and unstructured data. In NDP, queries can be executed at the storage layer.

[0043] As used herein, the term "user-defined function (UDF)" refers to any custom function that can perform preprocessing, runtime processing, and post-processing of data, including but not limited to common database operations (e.g., SQL operations). In the context of NDP, the concept of UDF is extended to encompass any external function that can be executed as part of an SQL operation or outside of an SQL operation, such as during the upload / download / access of certain objects in storage (e.g., during event-driven calls to the UDF).

[0044] As used herein, the term "directed acyclic graph (DAG)" refers to a graph based on the mathematical algorithms generated by the analytics platform for query planning. A DAG defines the order and content of execution steps, typically at the compute end. In the context of NDP, as discussed herein, certain parts of a DAG can be pushed down to be executed on the storage node, while the analytics platform can still execute the remaining parts of the DAG on the compute node and send the entire result back to the user after the storage node reports the partial DAG execution results.

[0045] As used herein, the term "acceleration" is associated with hardware acceleration, such as processor acceleration (e.g., using graphics processing units (GPUs) and single instruction multiple data (SIMD) CPUs) and software acceleration (e.g., using high-speed parsers). SIMD is a class of parallel computing processes commonly used to accelerate certain digital image / graphics processing or certain deep learning training processes that are also based on complex graphics processes such as neural networks. Acceleration can also be used for big data analytics.

[0046] As used in this article, the term "data skipping" refers to a technique that minimizes unnecessary data transfer from storage to computation. It typically uses data skipping indexes applied to structured datasets and stores summary metadata for each object in the dataset. For example, if a dataset has a temperature column, the summary metadata could include the lowest and highest temperatures among the objects. This summary metadata, much smaller than the data itself, can be indexed, so when a query looks for temperatures greater than 90°F, it can benefit from the index by skipping all objects whose metadata does not overlap with the predicate.

[0047] Publicly available technologies use Non-Deployment Processing (NDP) to push computational tasks from the compute network to the storage network, mitigating network bottlenecks associated with data processing in decomposed architectures. The benefits associated with NDP include: significantly reduced network traffic between compute and storage, particularly for highly selective compute requests; reduced storage input / output (I / O); faster overall processing time to better support service level agreement (SLA) goals and improve customer experience; assistance in mitigating the "too big to handle" problem for compute networks, such as those using an "ingest then compute" approach (which can lead to computational problems, unnecessary shuffling, memory shortages, and wasted computational resources if the ingested data becomes too large); full utilization of storage system resources as a complement to compute resources; reduced costs, as including more processing power in the storage system is cheaper than improving high-bandwidth network pipelines; and improved data privacy and oversight (e.g., NDP can shield sensitive information without data leaving the storage system).

[0048] Existing NDP technologies face several challenges in terms of both protocol and architecture. For example, they do not support complex query pushdown. Instead, they only support pushing down simple query operations from computing platforms like Spark and TensorFlow. This limitation stems primarily from the fact that these simple operations do not require further optimization, and storage systems can use simple query engines to serve NDP while supporting different computing platforms on the same storage system. In this respect, storage systems will struggle to optimize complex queries for each computing platform. Furthermore, existing data processing protocols do not push down application-specific and optimized task / query plans (e.g., in the form of DAGs) to the NDP's storage layer.

[0049] For big data analytics and artificial intelligence (BDA / AI), pushing UDFs down to the storage layer can be considered a crucial function because multiple operations that cannot be accomplished solely through queries (e.g., extract, transform, load (ETL) operations, machine learning (ML) operations, and deep learning (DL) operations) can be implemented within UDFs. Furthermore, modern computing platforms like Spark typically invoke UDFs in conjunction with queries during runtime. However, existing NDP technologies do not support UDF pushdown or queries processed by NDP within the storage layer.

[0050] Additional drawbacks of existing NDP technologies are related to their architectural implementation. More specifically, existing NDP technologies lack portability across different storage systems, leading to repetitive work in porting an NDP service from one type of data processing system to another. Furthermore, there is no universal architecture that allows NDP services to be executed on different storage system types, including serverless frameworks for UDF processing.

[0051] Compared to existing solutions, the exposed NDP technology associated with storage node-based NDP services can be used to perform complex SQL pushdown, standalone UDF pushdown, UDFs with SQL pushdown (e.g., via DAG pushdown), storage node-level hardware acceleration, storage I / O optimization using object indexes within the storage node, semantic awareness across multiple compute platforms, and execution of NDP using serverless frameworks as part of the NDP service. More specifically, semantic-aware protocols are used to send application-specific and optimized query / task plans from the DAG to the storage layer, enabling storage node-based NDP services to perform data processing functions to achieve application and workflow optimizations specific to the DAG. In this regard, multiple applications can run exposed NDP services with separately optimized execution plans (e.g., via DAG) on the same storage system. Exposed NDP services also support extended runtime and event-based UDF pushdown support for BDA and AI use cases.

[0052] The exposed NDP service uses different proxy modules to facilitate the processing of simple queries as well as DAGs, including complex queries and UDFs. For example, the exposed NDP service uses a storage request broker to redirect / intercept DAGs to a separate NDP service broker for processing. The NDP service can be deployed as part of a distributed storage node or externally (e.g., as an external NDP device within a storage network), making it highly portable and versatile. Furthermore, the NDP service supports various data processing technologies, including providing a serverless (or function-as-a-service, FaaS) framework for NDPs of UDFs, promoting distributed processing, parallelism, and tenant security.

[0053] Figure 1 A high-level system overview of a network architecture 100 using near data processing (NDP) services, providing some exemplary embodiments. (See references...) Figure 1Network architecture 100 may include multiple devices (e.g., user equipment) 102A, ..., 102N (collectively referred to as devices 102) communicatively coupled to network-based service infrastructure 114 via network 112. Devices 102A, ..., 102N are associated with corresponding users 106A, ..., 106N and can be used to interact with network-based service infrastructure 114 using network access clients (e.g., one of network access clients 104A, ..., 104N). Network access clients 104A, ..., 104N can be implemented as web clients or application (app) clients.

[0054] Users 106A, ..., 106N may generally be referred to as "a user" 106 or collectively as "users" 106. Each user 106 may be a human user (e.g., a human), a machine user (e.g., a computer configured by software programs to interact with device 102 and network-based service infrastructure 114), or any suitable combination thereof (e.g., a machine-assisted human or a machine supervised by a human). User 106 is not part of network architecture 100, but each user is associated with one or more devices 102 and may be a user of device 102 (e.g., user 106A may be the owner of device 102A, and user 106N may be the owner of device 102N). For example, device 102A may be a desktop computer, vehicle computer, tablet computer, navigation device, portable media device, or smartphone belonging to user 106A. Users 106A, ..., 106N may use devices 102A, ..., 102N to access services provided by network-based service infrastructure 114 (e.g., serverless computing services or FaaS, UDF processing service 122, DAG generation service 124, and NDP service 126). In this regard, user 106 may also be referred to as "client 106" or "tenant 106" of network-based service infrastructure 114. Serverless computing services may include instantiating and using virtual machines (VMs), virtual private clouds (VPCs), and application containers related to FaaS-related functions. In some embodiments, users 106A, ..., 106N may use devices 102A, ..., 102N to specify UDFs 108, ..., 110, which may be processed by UDF processing service 122 or used to generate one or more DAGs by DAG generation service 124.

[0055] The network-based service infrastructure 114 includes multiple computing devices 116, ..., 118, configured as distributed computing nodes forming a computing network and performing computational tasks, such as those associated with UDF processing service 122 and DAG generation service 124. The network-based service infrastructure 114 also includes multiple computing devices 120, ..., 122, configured as distributed storage nodes forming a storage network, wherein one or more distributed storage nodes (e.g., distributed storage node 120) are configured with NDP services.

[0056] UDF processing service 122 includes suitable circuitry, logic, interfaces, and / or code for processing UDFs 108, ..., 110 provided by users 106A, ..., 106N and received from devices 102A, ..., 102N via network 112. In some aspects, UDF processing service 122 may provide one or more UDFs 108, ..., 110 to a DAG generation service, such that the UDFs are included in one or more DAGs for execution by compute nodes or distributed storage nodes of the network-based service infrastructure 114. In an exemplary embodiment, UDFs 108, ..., 110 specify object keys and object location information (e.g., specifying a bucket or another object storage location for object storage), which are used to retrieve data objects (e.g., from object storage managed by one of the distributed storage nodes 120, ..., 122) prior to processing.

[0057] DAG generation service 124 includes suitable circuitry, logic, interfaces, and / or code for generating DAGs for processing by distributed computing nodes 116, ..., 118. In some embodiments, DAG generation service 124 generates DAGs as application-specific DAGs (e.g., based on an app executed on one of devices 102A, ..., 102N) for performing application-specific data processing tasks. In some aspects, the DAG includes queries and UDFs (e.g., one or more UDFs 108, ..., 110 received from devices 102A, ..., 102N). In an exemplary embodiment, the DAG (which may be a partial DAG) is pushed down to one of the distributed storage nodes for near-data processing (e.g., NDP services at distributed storage node 120). In some aspects, the computing network formed by distributed computing nodes 116, ..., 118 also includes a computing internal network 128 for internal communication between distributed computing nodes 116, ..., 118.

[0058] NDP service 126 includes suitable circuitry, logic, interfaces, and / or code, and is used to perform near-data processing using data processing instructions received from one of the distributed computing nodes 116, ..., 118. Although NDP service 126 in Figure 1 The diagram shows the process within a distributed storage node, but the invention is not limited thereto. More specifically, NDP service 126 can be implemented in a dedicated node within the storage network. The following is in conjunction with... Figures 2 to 9 Provide a more detailed description of NDP service 126.

[0059] In some aspects, the storage network formed by distributed storage nodes 120, ..., 122 also includes an internal storage network 130, which is used for internal communication between the distributed storage nodes 120, ..., 122. Additionally, the network-based service infrastructure 114 includes a compute-storage network 132, which is used to enable communication between the distributed compute nodes of the compute network and the distributed storage nodes of the storage network.

[0060] Network 112 can be any network that enables communication between or within machines, databases, and devices (e.g., devices 102A, ..., 102N and distributed computing nodes 116, ..., 118 within network-based service infrastructure 114). Therefore, network 112 can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. Network 112 may include one or more parts constituting a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0061] Figure 1 Any device shown herein can be implemented in a general-purpose computer, which is modified (e.g., configured or programmed) by software to become a special-purpose computer to perform the functions described herein for the machine, database, or device. As used herein, a “database” is a data storage resource that stores data structured as text files, tables, spreadsheets, relational databases (e.g., object-relational databases, NoSQL databases, web or graph databases), triple storage, hierarchical data storage, or any suitable combination thereof. Additionally, data accessed (or stored) via an application programming interface (API) or remote procedure call (RPC) can be considered as accessed from (or stored in) a database. Figure 1 Any two or more devices or databases shown may be combined into a single machine, database or device, and the functionality of any single machine, database or device described herein may be subdivided among multiple machines, databases or devices.

[0062] Figure 2 A block diagram illustrating a distributed storage node 212 with NDP service 220 within a network-based service infrastructure 200, providing some exemplary embodiments. (See reference...) Figure 2 Distributed storage nodes 212 in storage network 204 (e.g., Figure 1 One of the distributed storage nodes 120, ..., 122 in the network) and computing node 206 in the computing network 202 within the network-based service infrastructure 200 (e.g., Figure 1 (Communicate with one of the distributed computing nodes 116, ..., 118 in the system).

[0063] Distributed storage node 212 includes a storage request broker 214, a storage request gateway 216, an object service 218, an NDP service 220, and an object storage 242 storing data objects 246, ..., 248, as well as storage medium 244. The NDP service 220 includes an NDP service broker 222, a query service 224, a UDF processing service 226, a data index storage 234, and a hardware acceleration service 236. The data index storage 234 can be used in conjunction with data skipping operations. Specifically, the data index storage 234 includes indexes of data objects stored in the object storage 242, which can be used for fast data access and minimized storage I / O.

[0064] Storage request broker 214 analyzes data processing instructions (e.g., data processing instruction 207) received from compute node 206 and determines whether the data processing instruction should be processed by object service 218 or NDP service 220. For example, data processing instruction 207 may include multiple data processing operations, where at least one data processing operation includes a legacy data processing operation 208, such as a native operation. In some embodiments, legacy data processing operation 208 may include raw request operations, such as Amazon Simple Storage Service (Amazon S3) requests, which include fetch operations (e.g., reading a data object), place operations (e.g., storing a data object), delete operations, and paste operations (e.g., copying a data object) and / or non-complex query-based operations. In some aspects, non-complex query-based operations may include Amazon S3 selection operations, such as projection operations, filtering operations, and simple aggregation operations (e.g., SUM, AVERAGE, etc., without a GROUP BY operation).

[0065] In some embodiments, only raw request operations (e.g., raw S3 requests) are sent to object service 218 via storage request gateway 216, while any query-based operations and other types of requests and UDFs are sent to NDP service broker 222, so that such queries and UDFs can be processed using publicly available acceleration and optimization techniques used in NDP service 220.

[0066] Data processing instruction 207 may also include DAG 210. DAG 210 may include complex query-based operations, non-complex query-based operations, and UDF information associated with a UDF. An exemplary DAG is shown in... Figure 3 and Figure 4 The example UDF information that can be conveyed as part of a DAG is shown in [the diagram]. Figure 5 and Figure 6 As shown in the diagram. Non-complex query-based operations within DAG 210 can include Amazon S3 selection operations, such as data averaging, summing, grouping, etc. Complex query-based operations within DAG 210 can include grouping operations (e.g., GROUP BY), join operations, and limit operations. In this regard, when data processing instruction 207 includes legacy data processing operation 208, storage request broker 214 routes data processing instruction 207 to object service 218 for processing via storage request gateway 216. When data processing instruction includes DAG 210, storage request broker 214 routes data processing instruction 207 to NDP service 220. In some embodiments, data processing instruction 207 can include both legacy data processing operation 208 and DAG 210. In this case, storage request broker 214 routes legacy data processing operation 208 to storage request gateway 216 to communicate with object service 218, and routes DAG 210 to NDP service 220 for processing. After DAG 210 is processed by NDP service 220, DAG execution result 211 is generated and transmitted back to compute node 206.

[0067] NDP service broker 222 analyzes the received DAG 210 to determine whether it includes any complex query-based operations, non-complex query-based operations, and UDF information. Complex and non-complex query-based operations that are part of the DAG are routed to query service 224 for processing, while UDF information is routed to UDF processing service 226. UDF processing service 226 includes appropriate circuitry, logic, interfaces, and / or code for processing one or more UDFs based on the UDF information within DAG 210. In this regard, UDF processing service 226 uses at least one of a virtual machine (VM) 228, a container 230, or a serverless framework 232 to process the UDFs associated with DAG 210. Figure 6 Chinese combination Figure 5 A more detailed description of using the serverless framework 232 to handle UDFs is provided.

[0068] In an exemplary embodiment, legacy data processing operation 208 (which may include non-complex query-based operations) and complex query-based operations within DAG 210, along with UDF information including object key information and object location information, are used to access (or retrieve) data objects (e.g., one of data objects 246, ..., 248) from object storage 242 for processing. In some aspects, the NDP service 220's processing of data objects may utilize hardware acceleration provided by hardware acceleration service 236. More specifically, hardware acceleration service 236 may use SIMD CPU 238 and GPU 240 to accelerate certain data processing operations and improve the overall efficiency of NDP service 220.

[0069] Figure 3 A block diagram 300 illustrates, for example, the generation of an acyclic directed graph (DAG) 308 for an NDP in a distributed storage node. (See also: [link to block diagram]) Figure 3 In, for example Figure 2 The application request 302 is received at a computing node such as computing node 206. The application request 302 may include multiple application-specific instructions 304, including selection instructions, summation instructions, and grouping instructions. The application request 302 is processed by a query task analyzer 306, which may be, for example... Figure 1 This is part of the DAG generation service 124, etc. The DAG generation service generates a DAG 308 based on application-specific instructions 304, which are part of application request 302.

[0070] DAG 308 includes multiple data processing operations arranged within DAG 308 according to their execution order. For example, such as Figure 3 As shown, the data processing operations include a projection operation 310 performed first, followed by filter-aggregate operation pairs 312-318, 314-320, ..., 316-322. The generated DAG 308 is communicated to a storage node (e.g., distributed storage node 212) for near-data processing (e.g., performed by NDP service 220 in distributed storage node 212). After processing DAG 308, the DAG execution result is obtained and can be communicated back to the compute node that received application request 302.

[0071] Figure 4 A block diagram of an exemplary DAG 400 with user-defined functions (UDFs) is provided to illustrate some exemplary embodiments. (See reference...) Figure 4The DAG 400 can be generated by a DAG generation service 124 within a distributed computing node 116 of a network-based service infrastructure 114. The DAG 400 includes multiple data processing operations 402 to 418 arranged in execution order within the DAG. More specifically, operations 402, 404, and 406 can be executed concurrently with operations 408, 410, and 412. The results of data processing operations 406 and 412 are then used to execute sequential data processing operations 414, 416, and 418 to generate the DAG execution result.

[0072] In an exemplary embodiment, such as Figure 4 As shown, DAG 400 may include UDF information 412, which is passed by NDP service agent 222 to UDF processing service 226 for processing (e.g., as in...). Figure 6 Chinese combination Figure 5 (As described).

[0073] Figure 5 Some exemplary embodiments are shown by Figure 2 The NDP service is used for serverless framework 500 for UDF processing. (See reference) Figure 5 The serverless framework 500 can be the same as the serverless framework 232 used by the UDF processing service 226 within the NDP service 220.

[0074] During operation, the NDP service agent 502 (which can be used with...) Figure 2The NDP service agent 222 (same as the NDP service agent 222) detects UDF information 518 within the DAG received by the distributed storage node hosting the NDP service (e.g., distributed storage node 212). The UDF information may include UDF application code 519A, object key information 519B, and object location information 519C (including information identifying one or more object buckets or geological regions). The object key information 519B may include one or more security keys (or credentials) that can be used to authenticate access to data objects stored in the object storage. The UDF processing service 226 uses the object key information 519B and the object location information 519C to retrieve data objects (e.g., one of data objects 522, ..., 524) from the object storage 512. Additionally, the UDF processing service 226 uses the resources of the storage node to execute UDF application code 519A to determine one or more data processing functions of the UDF, packages / builds these functions into containers (Docker) (e.g., containers 1, ..., N), and then deploys these containers as workers 506, 508, ..., 510 of the serverless framework 500 to execute the UDF as a runtime query-based call to a serverless function. The serverless framework 500 also allocates resources, monitors the runtime status of the UDF's data processing functions, and deletes the containers once the corresponding tasks are completed. As used herein, the term "worker" is interchangeable with the term "function call." Once all container-based workers (or function calls) 506, ..., 510 have completed the UDF's data processing functions, a DAG execution result 526 is generated and communicated back to the compute node that initially communicated the DAG containing the UDF information 518.

[0075] In one exemplary embodiment, after receiving UDF information 518 and executing UDF application code 519A to obtain UDF data processing functionality, UDF processing service 226 can perform UDF data processing functionality using existing VM 228 or container 230. Compared to VM 228 and container 230 continuously running within distributed storage node 212, the serverless framework 232 will not use system resources without a data processing trigger, such as the received UDF information. In this regard, the serverless framework 232 deploys containers and uses them as workers 506, ..., 510, receiving only one UDF message and retrieving data objects using the UDF information. Additional functionality performed by the serverless framework 232 includes automation functions (e.g., the serverless framework 232 automatically manages the system resources of the container and manages the entire UDF lifecycle from building, deploying, servicing, and deleting the container for the purpose of performing UDF data processing functionality) and implementation functions (the serverless framework 232 can use container-based implementations and other mechanisms, such as web page assembly, etc.).

[0076] Unlike stateless, serverless computational functions, storage-side functions have associated state information (e.g., as used by BDA and AI applications). Since workers 506, ..., 510 do not communicate directly with each other, in an exemplary embodiment, shared storage locations within object storage 512 can be used to store intermediate function information 514 (e.g., state information conveyed by each container) via communication link 520. Additionally, communication link 520 can be used to exchange intermediate function information (or other state information or data) among workers 506, ..., 510.

[0077] In some embodiments, UDF application code 519A may be stored in function registry 516 within object store 512. In this case, UDF information 518 may include UDF call request 519D and function registry location 519E in place of UDF application code 519A. Serverless framework 500 uses function registry location 519E to obtain UDF application code from function registry 516 and continues to execute and deploy workers 506, ..., 510 to perform runtime query-based calls to the UDF.

[0078] In another embodiment, UDF processing service 226 can execute event-driven calls to the UDF. For example, UDF information 518 can specify an object storage event 519F for invoking data processing functions associated with the UDF. For example, object storage event 519F can include performing data manipulation procedures in object storage 512, such as storing new data objects or updating existing objects. Upon detecting an object storage event, serverless framework 500 can execute UDF application code to determine the data processing function and deploy workers 506, ..., 510 to perform the UDF's function.

[0079] Figure 6 A block diagram 600 illustrates query-based and event-driven invocation of a UDF within a distributed storage node, showcasing some exemplary embodiments. (See reference...) Figure 6 The DAG 602 with UDF information 604 is received by the UDF processing service 226 for further processing by the serverless framework 610. UDF information 604 includes UDF application code 606 and a UDF invocation request 608. UDF application code 606 executes by triggering the deployment of workers 506, ..., 510 based on the UDF invocation request 6082. In another embodiment, UDF application code 616 is stored in the function registry 614 of the object store 612. Based on the UDF invocation request 608, the serverless framework 610 can retrieve data from the function registry 614 (also in conjunction with...) Figure 5 The UDF application code 616 described is retrieved and executed to perform a runtime query-based call to the UDF as a serverless function.

[0080] In different embodiments, the UDF information specifies an object storage event (e.g., data storage event 620) as a call trigger for an event-driven invocation of the UDF as a serverless function. The UDF information 604 also specifies the location within the function registry 614 of object storage 612 where the UDF application code 616 of the UDF is stored. Upon detection of data storage event 620 (e.g., new data 618 stored in object storage 612), the UDF application code 616 is executed according to the function registry 614 to perform an event-driven invocation of the UDF as a serverless function.

[0081] Figure 7 A flowchart of method 700 for NDP within a distributed storage node suitable for a network architecture, with some exemplary embodiments provided. Method 700 includes operations 702, 704, 706, 708, and 710. By way of example and not limitation, method 700 is described as being performed by... Figure 2 NDP service 220 (also corresponding to Figure 8 NDP service 860 or Figure 9 The NDP service 960 is executed. At operation 702, data processing instructions are received from compute nodes of the network architecture at the distributed storage node. For example, data processing instruction 207 is received at distributed storage node 212 from compute node 206 of the network-based service infrastructure 200. At operation 704, a directed acyclic graph (DAG) is detected within the data processing instructions. For example, DAG 210 is detected within data processing instruction 207 received at distributed storage node 212. More specifically, storage request broker 214 may detect that data processing instruction 207 includes a DAG and pass the DAG to NDP service 220 for further processing. DAG 210 includes multiple data processing operations (e.g., ...). Figure 3 DAG 308 in the range of 310, ..., 322 Figure 4 The DAG 400 contains data processing operations 402, ..., 418, where multiple data processing operations are arranged within the DAG according to the execution order. The DAG 210 also includes user-defined function (UDF) information associated with UDFs (e.g., UDF information 518).

[0082] At operation 706, based on object key information within multiple data processing operations, at least a first data object is retrieved from the object storage of a distributed storage node (or one or more other distributed storage nodes in a storage network). For example, each data processing operation of the DAG may include object key information (e.g., object key information 519B) and object location information (e.g., object location information 519C), which can be used to retrieve a data object (e.g., data from object storage 512 or 242). Figure 5 Data objects 522, ..., 524 or Figure 2 (One of the data objects 246, ..., 248 in the data).

[0083] At operation 708, multiple data processing operations are performed using at least the retrieved first data object to execute a UDF and generate a DAG execution result, with the execution of the multiple data processing operations based on the execution order. For example, query service 224 can perform any complex query-based operation within DAG 210, and UDF processing service 226 can execute one or more functions associated with the UDF based on UDF information within DAG 210 to generate a DAG execution result (e.g., DAG execution result 211 or 526). At operation 710, in response to a data processing instruction, the DAG execution result is communicated to the compute node.

[0084] Figure 8 To illustrate a block diagram of a representative software architecture 800 of some exemplary embodiments, the software architecture can be used in conjunction with various device hardware described herein. Figure 8 This is merely a non-limiting example of software architecture 802; it should be understood that many other architectures can be implemented to facilitate the realization of the functionality described herein. Software architecture 802 can be implemented in, for example... Figure 9 The processing is performed on hardware such as a computing device 900, which, among other things, includes a processor 905, memory 910, storage devices 915 and 920, and I / O components (or interfaces) 925 and 930. A representative hardware layer 804 is shown, and the hardware layer can represent, for example... Figure 9 The computing device 900. A representative hardware layer 804 includes one or more processing units 806 having associated executable instructions 808. The executable instructions 808 represent executable instructions of the software architecture 802, including... Figures 1 to 7 The hardware layer 804 also includes memory and / or storage modules 810, which also have executable instructions 808. The hardware layer 804 may also include other hardware 812, which refers to any other hardware of the hardware layer 804, such as other hardware shown as part of the computing device 900.

[0085] existFigure 8 In the exemplary architecture, software architecture 802 can be conceptualized as a stack of layers, each providing specific functionality. For example, software architecture 802 may include layers such as operating system 814, libraries 816, framework / middleware 818, application 820, and presentation layer 844. Operationally, application 820 and / or other components within each layer can call application programming interface (API) calls 824 through the software stack and, in response to API call 824, receive responses, return values, etc., as shown in message 826. Figure 8 The layers shown are representative, and not all software architectures 802 have all layers. For example, some mobile or dedicated operating systems may not provide a framework / middleware 818, while other operating systems may provide such layers. Other software architectures may include additional or different layers.

[0086] Operating system 814 can manage hardware resources and provide public services. Operating system 814 may include, for example, a kernel 828, services 830, and drivers 832. Kernel 828 can act as an abstraction layer between hardware and other software layers. For example, kernel 828 can be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 830 can provide other public services to other software layers. Drivers 832 can be responsible for controlling or connecting to the underlying hardware. For example, depending on the hardware configuration, drivers 832 may include display drivers, camera drivers, etc. Drivers, flash drives, serial communication drivers (e.g., universal serial bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0087] Library 816 can provide common infrastructure that can be utilized by application 820 and / or other components and / or layers. Library 816 typically provides functionality that allows other software modules to perform tasks more efficiently than by directly connecting to the underlying operating system 814 functions (e.g., kernel 828, services 830, and drivers 832). Library 816 may include system libraries 834 (e.g., the C standard library), which can provide functions such as memory allocation, string manipulation, and mathematical functions. Furthermore, library 816 may include API libraries 836, such as media libraries (e.g., libraries supporting the representation and manipulation of various media formats (e.g., MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., OpenGL frameworks for rendering 2D and 3D graphics content on a display), database libraries (e.g., SQLite providing various relational database functions), web page libraries (e.g., WebKit providing web browsing functionality), etc. Library 816 may also include various other libraries 838 to provide many other APIs to application 820 and other software components / modules.

[0088] Frameworks / middleware 818 (sometimes also called middleware) provide advanced common infrastructure that applications 820 and / or other software components / modules can use. For example, frameworks / middleware 818 can provide various graphical user interface (GUI) functionalities, advanced resource management, advanced location services, etc. Frameworks / middleware 818 can provide a wide variety of other APIs that applications 820 and / or other software components / modules can use, some of which may be specific to a particular operating system 814 or platform.

[0089] Application 820 includes built-in application 840 and / or third-party application 842. Examples of representative built-in applications 840 may include, but are not limited to, contact applications, browser applications, reader applications, location applications, media applications, communication applications, and / or game applications. Third-party applications 842 may include any built-in application 840 as well as a wide variety of other applications. In a specific example, third-party applications 842 (e.g., those used by entities other than platform-specific vendors using Android) TM Or iOS TM Applications developed using a software development kit (SDK) can be used on iOS. TM Android TM , Mobile software running on a mobile operating system such as a phone or other mobile operating system. In this example, a third-party application 842 may call API calls 824 provided by the mobile operating system (e.g., operating system 814) to facilitate the implementation of the functionality described herein.

[0090] In some embodiments, application 820 further includes NDP service 860, which may be the same as NDP service 220 discussed herein. In another embodiment, NDP service 860 may be implemented as part of operating system 814.

[0091] Application 820 can utilize built-in operating system functions (e.g., kernel 828, services 830, and drivers 832), libraries (e.g., system library 834, API library 836, and other libraries 838), and frameworks / middleware 818 to create user interfaces for interacting with the system user. Alternatively or additionally, in some systems, interaction with the user can be achieved through a presentation layer (e.g., presentation layer 844). In these systems, the application / module "logic" can be independent of the aspects of the application / module interacting with the user.

[0092] Some software architectures utilize virtual machines. Figure 8 In the example, the virtual machine is shown as virtual machine 848. The virtual machine creates a software environment where applications / modules can function as if they were on a hardware machine (e.g., ...). Figure 9 The virtual machine 848 executes the same operation on the host operating system (900). Figure 8 The virtual machine 848 is hosted by an operating system 814 and typically, but not always, has a virtual machine monitor 846 that manages the operation of the virtual machine 848 and its connection to the host operating system (i.e., operating system 814). Software architecture 802 executes within the virtual machine 848, such as operating system 850, libraries 852, frameworks / middleware 854, applications 856, and / or presentation layers 858. These software architecture layers executing within the virtual machine 848 may be the same as or different from their corresponding counterparts described above.

[0093] Figure 9 The diagram illustrates the circuitry of a device for implementing algorithms and methods of execution in some exemplary embodiments. Not all components need to be used in all embodiments. For example, clients, servers, and cloud-based network devices may each use different sets of components, or, in the case of a server, a larger storage device may be used.

[0094] An exemplary computing device in the form of a computer 900 (also referred to as a computing device 900, computer system 900, or computer 900) may include a processor 905, memory 910, removable memory 915, non-removable memory 920, an input interface 925, an output interface 930, and a communication interface 935, all of which are connected via a bus 940. Although the exemplary computing device is shown and described as a computer 900, the computing device may take different forms in different embodiments.

[0095] Memory 910 may include volatile memory 945 and non-volatile memory 950, and may store program 955. Computer 900 may include or have access to a computing environment, which includes a variety of computer-readable media, such as volatile memory 945, non-volatile memory 950, removable memory 915, and non-removable memory 920. Computer memory includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), digital versatile disk (DVD) or other optical disc storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0096] Computer-readable instructions stored on a computer-readable medium (e.g., program 955 stored in memory 910) can be executed by processor 905 of computer 900. Hard disk drives, CD-ROMs, and RAM are some examples of articles of art that include non-transitory computer-readable media such as storage devices. The terms "computer-readable medium" and "storage device" do not include carrier waves considered too transient. "Computer-readable non-transitory medium" includes all types of computer-readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media. It should be understood that software can be installed in a computer and sold with the computer. Alternatively, software can be obtained and loaded into a computer, including obtaining software through physical media or distribution systems, including, for example, from a server owned by the software author or from a server not owned by the software author but used. For example, software can be stored on a server for distribution over a network. As used herein, the terms "computer-readable medium" and "machine-readable medium" are interchangeable.

[0097] Program 955 may utilize the services discussed herein, such as NDP service 960. In some respects, NDP service 960 may be identical to NDP service 220 discussed herein.

[0098] Any one or more modules described herein can be implemented using hardware such as a machine's processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or any suitable combination thereof. Furthermore, any two or more of these modules can be combined into a single module, and the functionality of a single module described herein can be subdivided among multiple modules. Moreover, modules described herein as being implemented within a single machine, database, or device can be distributed across multiple machines, databases, or devices, according to various exemplary embodiments.

[0099] In some aspects, one or more modules (e.g., services) within the NDP service 960 can be integrated into a single module, performing the corresponding functions of the integrated module. In other words, the NDP service can be implemented as a single service on a distributed storage node (e.g., using resources such as virtual machines or containers), or as an NDP service (e.g., such as...). Figure 2 Individual components (e.g., modules or services) shown in the diagram can be implemented separately (e.g., at the same or different storage nodes).

[0100] Although several embodiments have been described in detail above, other modifications are possible. For example, the logical flow depicted in the figures does not require the specific or sequential order shown to achieve the desired result. Other steps may be provided, steps may be removed from the described flow, and other components may be added to or removed from the described system. Other embodiments are within the scope of the appended claims.

[0101] It should also be understood that software comprising one or more computer-executable instructions may be installed in and sold with one or more computing devices consistent with the present invention, said computer-executable instructions facilitating processing and operation of any or all of the steps relating to the present invention as described above. Alternatively, the software may be obtained and loaded into one or more computing devices, including obtaining the software via physical media or a distribution system, including, for example, obtaining the software from a server owned by the software author or from a server not owned by the software author but used. For example, the software may be stored on a server for distribution over a network.

[0102] Furthermore, those skilled in the art will understand that the invention, in its application, is not limited to the details of the construction and arrangement of the components set forth in the description or shown in the drawings. The embodiments described herein can have other embodiments and can be practiced or performed in various ways. Furthermore, it should be understood that the wording and terminology used herein are descriptive and should not be considered limiting. The use of “including,” “comprising,” or “having,” and variations thereof herein is intended to cover the items listed thereafter and their equivalents, as well as additional items. Unless otherwise limited, the terms “connected,” “coupled,” and “mounted,” and variations thereof, are used extensively herein and cover direct and indirect connections, couplings, and mountings. Furthermore, the terms “connected” and “coupled,” and variations thereof, are not limited to physical or mechanical connections or couplings. Additionally, terms such as “upper,” “lower,” “bottom,” and “top” are relative and used for illustrative purposes but are not limiting.

[0103] The components of the illustrative devices, systems, and methods employed according to the illustrated embodiments may be implemented at least in part in digital electronic circuits, analog electronic circuits, or computer hardware, firmware, software, or combinations thereof. For example, these components may be implemented as computer program products (e.g., computer programs, program code, or computer instructions) tangibly embodied in an information carrier or in a machine-readable storage device for execution by a data processing apparatus (e.g., a programmable processor, computer, or multiple computers) or for controlling the operation of the data processing apparatus.

[0104] Computer programs can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs can be deployed to execute in one or more computers at a single site, or distributed across multiple sites interconnected via a communication network. Furthermore, the functional programs, code, and code segments used to implement the techniques described herein are readily understood by a programmer in the art to which the techniques described herein belong to be within the scope of the claims. The method steps associated with the illustrative embodiments can be executed by one or more programmable processors to perform functions (e.g., manipulating input data and / or generating output). For example, the method steps can also be executed by special-purpose logic circuitry (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), and the means for performing the method can be implemented as said special-purpose logic circuitry.

[0105] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein can be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, it may also be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors coupled to a DSP core, or any other similar configuration.

[0106] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. Essential components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to one or more mass storage devices for storing data to receive data from and / or transfer data to said mass storage device. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices, such as electrically programmable read-only memory or electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, and data storage disks (e.g., magnetic disks, internal hard disks, or removable disks, magneto-optical disks, CD-ROMs, and DVD-ROMs). Processors and memory can be supplemented by or incorporated into dedicated logic circuits.

[0107] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0108] As used herein, “machine-readable medium” (or “computer-readable medium”) means a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of memory (e.g., erasable programmable read-only memory (EEPROM)) and / or any suitable combination thereof. The term “machine-readable medium” should be understood to include a single medium or multiple media capable of storing processor instructions (e.g., a centralized or distributed database, or associated caches and servers). The term “machine-readable medium” should also be understood to include any medium (or combination of media) capable of storing instructions executable by one or more processors 905, such that, when executed by one or more processors 905, the instructions cause one or more processors 905 to perform any or more methods described herein. Accordingly, “machine-readable medium” refers to a single storage device or apparatus, and a “cloud-based” storage system or storage network comprising multiple storage devices or apparatuses. As used in this article, the term "machine-readable medium" does not include the signal itself.

[0109] Furthermore, without departing from the scope of the invention, the technologies, systems, subsystems, and methods described and illustrated as discrete or separate in the various embodiments can be combined or integrated with other systems, modules, technologies, or methods. Other items shown or described as coupled to each other, or directly coupled, or communicating with each other, may be indirectly coupled or communicated through some interface, device, or intermediate component, electrically, mechanically, or otherwise. Other examples of changes, substitutions, or alterations can be determined by those skilled in the art and may be exemplified without departing from the scope of the disclosure herein.

[0110] Although the invention has been described with respect to specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made without departing from the scope of the invention. For example, other components may be added to or removed from the described system. Therefore, the specification and drawings are to be regarded only as illustrative of the invention as defined by the appended claims, and are intended to cover any modifications, variations, combinations, or equivalents falling within the scope of the invention. Other aspects may be within the scope of the appended claims.

Claims

1. A computer implementation method for near data processing (NDP) within a distributed storage node in a network architecture, characterized in that, The method includes: The distributed storage node receives data processing instructions from the computing nodes of the network architecture. Within the data processing instructions, a directed acyclic graph (DAG) is detected, wherein the DAG includes multiple data processing operations arranged in the DAG according to the execution order, and includes UDF information associated with user-defined functions (UDFs). The first data object is retrieved from the object storage of the distributed storage node based on the object key information within the multiple data processing operations. The plurality of data processing operations are performed using the retrieved first data object to execute the UDF and generate a DAG execution result, wherein the execution of the plurality of data processing operations is based on the execution order; In response to the data processing instruction, the execution result of the DAG is transmitted to the computing node.

2. The computer implementation method according to claim 1, characterized in that, Also includes: The data processing instructions are detected to include the DAG and legacy data processing operations; The legacy data processing operation is routed to the object service of the distributed storage node to be executed using a second data object from the object storage, the second data object being selected based on a second object key information within the legacy data processing operation.

3. The computer implementation method according to claim 2, characterized in that, The legacy data processing operations include non-complex query-based operations or raw request operations. The raw request operations include at least one of the following operations: get operation, place operation, paste operation, and delete operation. The non-complex query-based operations include at least one of the following operations: filter operation and aggregation operation.

4. The computer implementation method according to claim 2, characterized in that, Also includes: The DAG is routed to the NDP service of the distributed storage node, the NDP service being configured to configure the virtual resources of the distributed storage node to perform the retrieval of at least the first data object and the execution of the plurality of data processing operations using the retrieved first data object.

5. The computer implementation method according to claim 1, characterized in that, The multiple data processing operations include complex query-based operations, which further include the object key information and the object location information of the storage bucket where the first data object is stored.

6. The computer implementation method according to claim 1, characterized in that, The UDF information includes the UDF application code of the UDF and the object key information.

7. The computer implementation method according to claim 6, characterized in that, The computer implementation method further includes: The UDF application code is executed using the serverless framework of the distributed storage node to execute the UDF as a runtime query-based call to a serverless function using the first data object associated with the object key information.

8. The computer implementation method according to claim 7, characterized in that, The runtime query-based invocation for executing the UDF also includes: Multiple container-based function calls are deployed within the serverless framework of the distributed storage node to execute the UDF.

9. The computer implementation method according to claim 8, characterized in that, The plurality of container-based function calls within the serverless framework of the distributed storage node are used to exchange state information through shared storage within the object storage of the distributed storage node.

10. The computer implementation method according to claim 6, characterized in that, The computer implementation method further includes: The UDF application code is executed using a virtual machine or container instantiated within the distributed storage node.

11. The computer implementation method according to claim 1, characterized in that, The UDF information includes a request to invoke the UDF on the first data object. The UDF information also specifies the location of the UDF application code stored in the function registry where the object is stored. The computer implementation method further includes: The UDF application code is executed according to the function registry to perform a runtime query-based call to the UDF as a serverless function.

12. The computer implementation method according to claim 6, characterized in that, The UDF information specifies an object storage event as a call trigger for event-driven invocation of the UDF as a serverless function. The UDF information also specifies the location within the object storage function registry where the UDF application code of the UDF is stored. The computer implementation method further includes: Detect the object storage event specified by the UDF information, the object storage event being associated with data updates in the object storage; Based on the detection of the object storage event, the UDF application code is executed according to the function registry to execute the event-driven call of the UDF as a serverless function.

13. A system, characterized in that, include: Memory, storing instructions; One or more processors communicate with the memory, wherein the one or more processors execute the instructions to: Receive data processing instructions from the compute nodes in the network architecture at the distributed storage nodes; Within the data processing instructions, a directed acyclic graph (DAG) is detected, wherein the DAG includes multiple data processing operations arranged in the DAG according to the execution order, and includes UDF information associated with user-defined functions (UDFs). The first data object is retrieved from the object storage of the distributed storage node based on the object key information within the multiple data processing operations. The plurality of data processing operations are performed using the retrieved first data object to execute the UDF and generate a DAG execution result, wherein the execution of the plurality of data processing operations is based on the execution order; In response to the data processing instruction, the execution result of the DAG is transmitted to the computing node.

14. The system according to claim 13, characterized in that, The UDF information includes the UDF application code and the object key information, and the one or more processors execute the instructions to: The UDF application code is executed using the serverless framework of the distributed storage node to execute the UDF as a runtime query-based call to a serverless function using the first data object associated with the object key information.

15. The system according to claim 14, characterized in that, In order to execute the runtime query-based call of the UDF as a serverless function, the one or more processors execute the instructions to: Multiple container-based function calls are deployed within the serverless framework of the distributed storage node to execute the UDF.

16. The system according to claim 15, characterized in that, The plurality of container-based function calls within the serverless framework of the distributed storage node are used to exchange state information through shared storage within the object storage of the distributed storage node.

17. A computer-readable medium storing computer instructions for near data processing (NDP) within a distributed storage node of a network architecture, characterized in that, When executed by one or more processors, the instructions cause the one or more processors to perform the following operations: The distributed storage node receives data processing instructions from the computing nodes of the network architecture. Within the data processing instructions, a directed acyclic graph (DAG) is detected, wherein the DAG includes multiple data processing operations arranged in the DAG according to the execution order, and includes UDF information associated with user-defined functions (UDFs). The first data object is retrieved from the object storage of the distributed storage node based on the object key information within the multiple data processing operations. The plurality of data processing operations are performed using the retrieved first data object to execute the UDF and generate a DAG execution result, wherein the execution of the plurality of data processing operations is based on the execution order; In response to the data processing instruction, the execution result of the DAG is transmitted to the computing node.

18. The computer-readable medium according to claim 17, characterized in that, The UDF information includes the UDF application code of the UDF and the object key information, and the instructions also cause the one or more processors to perform operations including the following: The UDF application code is executed using a virtual machine or container instantiated within the distributed storage node.

19. The computer-readable medium according to claim 17, characterized in that, The UDF information includes a request to invoke the UDF on the first data object. The UDF information also specifies the location of the UDF application code stored in the function registry of the object. The instruction further causes the one or more processors to perform operations including the following: The UDF application code is executed according to the function registry to perform a runtime query-based call to the UDF as a serverless function.

20. The computer-readable medium according to claim 17, characterized in that, The UDF information specifies an object storage event as a call trigger for an event-driven invocation of the UDF as a serverless function. The UDF information also specifies the location within the object storage function registry where the UDF application code of the UDF is stored. The instructions further cause the one or more processors to perform operations including: Detect the object storage event specified by the UDF information, the object storage event being associated with data updates in the object storage; Based on the detection of the object storage event, the UDF application code is executed according to the function registry to execute the event-driven call of the UDF as a serverless function.

Images