Distributed computing system and method for data processing and storage medium

Abstract

This application relates to distributed computing systems and methods for data processing, as well as storage media. Generally, this disclosure describes techniques for applying a distributed pipeline model in a distributed computing system so that the processing nodes of the distributed computing system process data to perform tasks according to a distributed pipeline having an execution topology specified within a pipeline statement.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 049,920, filed July 9, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to a computing system, and more particularly, to distributed processing within the computing system. Background Technology

[0003] Nodes running on a computing device can interconnect to form a networked distributed computing system to exchange data and share resources. In some examples, multiple nodes running on a computing device are interconnected to jointly execute one or more applications to perform jobs. Nodes can include bare-metal servers, virtual machines, containers, processes, and / or other execution elements with data processing capabilities for the distributed computing system. Each node can independently perform various operations for the distributed computing system, such as collecting, processing, and exporting data, and nodes can communicate with each other to distribute the processed data. Summary of the Invention

[0004] Generally, this disclosure describes techniques for applying a distributed pipeline model in a distributed computing system so that processing nodes of the distributed computing system process data to perform tasks according to a distributed pipeline having an execution topology specified within a pipeline statement. For example, a computing device may receive a pipeline statement for a task. The pipeline statement includes multiple stage statements, each stage statement describing a corresponding stage of a set of one or more operations to be performed. One or more stage statements also specify topology information for the corresponding stage. For example, a stage statement may specify that the corresponding stage includes a sub-pipeline to be performed by one or more specified processing nodes of the distributed computing system. The stage statement may specify that the specified one or more processing nodes perform one or more operations of the sub-pipeline. In some cases, the stage statement also specifies a next stage that will receive the output of the corresponding stage for the stage statement. In some cases, the pipeline statement is human-readable text to allow operators to easily arrange operations within stages and arrange stages within the pipeline statement to specify a distributed pipeline having an overall execution topology for the tasks performed by the distributed computing system. The computing device receiving the pipeline statement may be, or can be, one of the processing nodes of the distributed computing system. In this context, the processing node can be referred to as the origin.

[0005] Computing devices process pipelined statements to cause processing nodes in a distributed computing system to process data according to the execution topology specified within the pipelined statement. For example, the computing device may output individual commands, at least some of the pipelined statements, stage information, configuration data, or other control information to the processing nodes to cause the nodes to process data according to the execution topology. The processing nodes then begin executing operators for the operations within the stage, binding the operators together using input / output channels such as standard streams (stdin, stdout) or queues. For example, operators can be processes, and operators bound within a stage can be referred to as subpipelines. If a stage statement specifies the next stage as the final operation of the corresponding stage, one or more processing nodes executing the corresponding stage can bind the final operation of that stage to one or more processing nodes configured to execute the next stage. Binding multiple stages between processing nodes may include creating communication channels between the processing nodes that operate over a network such as sockets. The node binding from one stage to the next stage allows data output from the final operation of that stage to be either fan-out (i.e., output by one processing node and received by multiple processing nodes) or fan-in (i.e., output by multiple processing nodes and received by one processing node) to the next stage.

[0006] In one example, a pipeline statement may include a first stage and a second stage. The first stage specifies one or more operations to be processed by each of multiple processing nodes, and the second stage specifies one or more operations to be processed by fan-in nodes (such as the origin) among the processing nodes. Multiple processing nodes can generate a first sub-pipeline for the first stage, which binds the operators of the first stage to process data to perform the corresponding operations, and then sends the results of the operators to the second stage. As an example, each processing node can orchestrate queues and interfaces for its local operators and generate send operations to send the results of its local operators to the fan-in nodes. The fan-in nodes can generate a second sub-pipeline for the second stage, which binds the operators of the second stage and receives the results of the operations of the first stage output by the multiple processing nodes. As an example, the fan-in node can orchestrate queues and interfaces for its local operators to first receive the results of the operations of the first stage and also execute its local operators to perform the operations of the second stage.

[0007] The techniques described herein can provide one or more technical advantages that enable at least one practical application. For example, the techniques can extend the traditional single-device Unix-style pipelining model to implement a distributed pipelining model, in which the processing nodes of a distributed computing system independently set up sub-pipelines for the overall distributed pipeline at least partially, and process data according to the execution topology specified in the pipelining statement to execute tasks. By automatically generating distributed pipelines to bind operations to be processed according to the topology, the techniques allow users to specify all operations and topologies for the distributed pipeline in the form of relatively simple pipelining statements, thus avoiding the typically complex and tedious task of configuring not only each node of the distributed computing system with the operations to be executed, but also with the inter-node communication channels of the topology between nodes. The specified topology can leverage the parallelism within and between processing nodes of the distributed computing system to execute operations of sub-pipelines defined for stages in the pipelining statement at least partially concurrently.

[0008] In some examples, a distributed computing system includes multiple computing devices configured to receive pipeline statements. The pipeline statements include a first-stage statement defining a first stage including a first sub-pipeline and specifying a second stage as the recipient of the result data of the first sub-pipeline. The pipeline statements also include a second-stage statement defining the second stage as including a second sub-pipeline. The multiple computing devices include a first processing node configured to configure the first sub-pipeline in the first processing node, at least based on a first pipeline setup specification generated by the processing pipeline statements, to send the result data of the first sub-pipeline in the first processing node to a second processing node. The multiple computing devices also include a second processing node configured to configure the second sub-pipeline in the second processing node, at least based on a second pipeline setup specification generated by the processing pipeline statements, and to input the result data of the first sub-pipeline in the first processing node into the second sub-pipeline in the second processing node.

[0009] In some examples, a distributed computing system includes multiple devices configured to execute corresponding cooperative programs to: receive pipeline statements, wherein the pipeline statements include multiple stages, wherein each stage includes one or more operations to be processed by one or more of the multiple devices, and wherein one or more operations of a stage will be processed in different topologies, and generate sub-pipelines of the distributed pipeline to bind one or more operations of a stage and output result data from the final stage of the stage.

[0010] In some examples, the method includes: receiving a command comprising multiple stages by a device among a plurality of devices connected via a network, wherein each stage includes one or more operations to be processed by one or more devices among the plurality of devices, and wherein the one or more operations in the stage will be processed in different topologies; the device distributing the command to other devices among the plurality of devices, such that the other devices among the plurality of devices each generate a first sub-pipeline of a distributed pipeline, wherein the first sub-pipeline is bound to one or more operations processed in a first topology of different topologies; and the device generating a second sub-pipeline of a distributed pipeline, wherein the second sub-pipeline is bound to one or more operations processed in a second topology of different topologies.

[0011] In some examples, the method includes: obtaining a pipeline statement from a distributed computing system including a first processing node and a second processing node, the pipeline statement including a first stage statement defining a first stage including a first sub-pipeline and designating a second stage as a recipient of the result data of the first sub-pipeline, the pipeline statement also including a second stage statement defining a second stage statement including a second sub-pipeline; configuring the first sub-pipeline in the first processing node to send the result data of the first sub-pipeline in the first processing node to the second processing node, at least based on a first pipeline setup specification generated from the pipeline statement; and configuring the second sub-pipeline in the second processing node to input the result data of the first sub-pipeline in the first processing node to the second sub-pipeline in the second processing node, at least based on a second pipeline setup specification generated from the pipeline statement.

[0012] Details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, the accompanying drawings, and the technical solutions. Attached Figure Description

[0013] Figure 1 This is a block diagram of an example distributed computing system 2, which includes multiple processing nodes configured to execute a distributed pipeline generated from pipeline statements according to one or more aspects of the disclosed technology.

[0014] Figures 2A-2B This is a block diagram of an example subpipeline of a distributed pipeline based on one or more aspects of the technology in this disclosure.

[0015] Figures 3A-3C The block diagram illustrates an example implementation of a distributed pipeline in more detail based on one or more aspects of the disclosed technology.

[0016] Figure 4It is a flowchart illustrating an example operation of a distributed system based on one or more aspects of the disclosed technology.

[0017] Figure 5 This is a block diagram of an example computing device operating according to one or more techniques disclosed herein.

[0018] Throughout the accompanying drawings and text, the same reference characters refer to the same elements. Detailed Implementation

[0019] Figure 1 This is a block diagram of an example distributed computing system 2, which includes multiple processing nodes configured to execute a distributed pipeline generated from pipeline statements according to one or more aspects of the disclosed techniques. The distributed pipeline includes operations bound and processed by the processing nodes according to the topology indicated in the pipeline statements.

[0020] exist Figure 1 In the example, the distributed computing system 2 includes multiple computing devices 6A-6N (collectively referred to as "Device 6") coupled to and communicating via network 10. Network 10 can be a local area network ("LAN"), a wide area network ("WAN"), a metropolitan area network ("MAN"), or other types of network. Device 6 of network 10 can exchange data using one or more packet-based protocols, such as Transmission Control Protocol (TCP) / Internet Protocol (IP).

[0021] Each of the devices 6 is a real or virtual processing node and performs or represents at least one processing node 8 within a distributed computing system (sometimes more simply referred to as a “distributed system”). Each of the devices 6 can be, for example, a workstation, laptop computer, tablet computer, rack server, other real or virtual server, container, or virtual machine. In some cases, any of the devices 6 can represent “Internet of Things” (IoT) devices, such as cameras, sensors, televisions, appliances, etc. As an example implementation, devices 6 can represent devices in a smart city grid, where each device is configured to collect and / or process data. In this example, devices 6B-6N can represent sensor devices (e.g., smart meters) to collect and / or process data, and device 6A can represent a device including sensor devices and a central processing unit (“CPU”) to append, process, and / or derive data. In some examples of the distributed computing system 2, devices 6 can be heterogeneous, as devices 6 can represent devices with different hardware and / or software. For example, device 6A can be a desktop workstation, while device 6B can be a rack server in a cloud computing center, and device 6C can be a wireless sensor, and so on. The hardware environment of device 6 can also be heterogeneous. For example, device 6 may include any one or more CPUs with different numbers of processing cores, ASICs, field-programmable devices, graphics processing units (GPUs), or other processing circuitry.

[0022] In this example, devices 6A-6N each include processing nodes 8A-8N (collectively referred to as "processing nodes 8"). Processing nodes 8 execute programs 9A-9N (collectively referred to as "programs 9") to process commands. Commands may represent partial execution steps / processes used to perform an overall task. Commands are included in pipeline statements 20 and, in some cases, may be text-based. The task of the distributed computing system 2 may include one or more operations corresponding to commands, such as retrieving data, processing data, exporting data, etc. Commands may refer to a specific program of program 9 executed by each processing node 8. In some examples, each processing node 8 represents an in-memory storage engine that manages data stored in a shared memory segment or partition, which is directly accessible to the storage engine. In some cases, a single device 6 may execute multiple instances of processing nodes 8, for example, as multiple virtual machines. However, in the case of a bare-metal server, for example, one of devices 6 may be one of processing nodes 8, because the device executes program 9 directly on the hardware.

[0023] Program 9 may include a command-line interface (CLI), a graphical user interface (GUI), or other interfaces for receiving and processing text-based commands, such as a shell. Program 9 may alternatively or additionally include a series of different processes, applications, scripts, or other executable programs that can be grouped together to perform a larger task. As an example, program 9A may be arranged within device 8A to implement a larger task specified using pipelined execution, following a single-device Unix-style pipelined model, where programs 9A are linked together through their standard streams so that the output data of one program 9A is routed as input data to the input of another program 9A that processes the input data. The output data of that program is routed as input data to the input of another program 9A, and so on, until the final output data of the last program 9A in the pipeline (e.g., to an interface, another program, or memory). Program 9 may also refer to other sets of different routines, functions, modules, or operations executed by one or more processes. For example, program 9A can refer to a single shell executed by processing node 8A, which can receive one or more operators arranged in the pipeline, execute the corresponding operations, and arrange the flow of these operations to implement the pipeline.

[0024] According to the techniques described herein, processing node 8 collaboratively executes program 9 based on a distributed pipeline generated from pipeline statement 20. The distributed pipeline includes programs 9A-9N arranged in a pipeline topology, which may include intra-process node branches and inter-process node branches. In the example shown, the pipeline topology includes... Figure 1 The inter-process node branches shown are inter-process node pipelines that perform one or more operations bound to different processing nodes.

[0025] exist Figure 1 In the example, user 4 can provide pipeline statement 20 to program 9A running on processing node 8A of device 6A. The device that initially receives commands from user 4 is referred to herein as the "source device". Pipeline statement 20 can specify tasks to be jointly processed by device 6, such as reading data, filtering, transforming, mapping, analyzing or processing data, sorting data, exporting data, or other operations.

[0026] Pipeline statement 20 specifies multiple stage statements, each stage statement describing a corresponding stage for a set of one or more operations to be performed. One or more stage statements also specify topology information for the corresponding stage. For example, a stage statement may specify that the corresponding stage includes one or more specified nodes 8 of the distributed system. A stage statement may specify that the specified one or more nodes 8 will perform one or more specified operations. In some cases, a stage statement also specifies a next stage that will receive the output of the corresponding stage of the stage statement. Pipeline statement 20 can be human-readable text to allow user 4 to easily arrange operations within stages and arrange stages within pipeline statement 20 to specify the overall execution topology of the tasks performed by the distributed computing system 2.

[0027] The node that receives pipeline statement 20 can be one of the processing nodes 8, or can execute it. Figure 1 In the example, if the node receiving pipeline statement 20 is one of the processing nodes 8, that node (processing node 8A in the example shown) can be referred to as the origin node, or more simply, the origin. In some cases, a separate device and / or process that is not one of the processing nodes 8 can receive and process pipeline statement 20. In this case, the separate device and / or process can be referred to as the controller, and the controller can designate one of the processing nodes 8 as the origin. Typically, the origin generates and outputs the final result of processing pipeline statement 20. However, this output can be via the controller. The origin or controller represents a compiler computing device that receives the pipeline statement and performs the operations described below to compile the pipeline statement to generate and subsequently distribute a pipeline setup specification, which includes at least partial topology information for configuring a distributed processing topology in and between processing nodes 8 for the overall task represented in pipeline statement 20. However, this compiler computing device and operations can be performed by multiple different computing devices. In the following text, these operations are described relative to the origin of receiving pipeline statement 20, but at least in some respects, these operations can be performed by a controller that delegates some operations to one of the processing nodes 8.

[0028] Processing node 8A receives pipeline statement 20. In response, processing node 8A processes pipeline statement 20 to determine, based on the syntax of pipeline statement 20, that one or more operations of pipeline statement 20 will be processed on different processing nodes 8. For example, the syntax of a pipeline statement may include multiple stage statements, each stage statement specifying one or more operations for a corresponding stage. Figure 1In the example, pipeline statement 20 may include one or more operations to be processed by each of the processing nodes 8A-8N as part of a first stage (e.g., data collection and processing), and one or more operations to be processed by the origin processing node 8A as part of a second stage (e.g., data aggregation and export).

[0029] Pipeline statement 20 can provide topology information through its structure, syntax, and semantics, which describes the configuration, partition cardinality, and operator ordering abstracted as patterns of the cluster of processing nodes 8 or other distributed meshes. The distributed pipeline described by pipeline statement 20 can be non-linear, cyclic, implement feedback loops, and / or be based on the capabilities of the processing nodes (e.g., available hardware and / or software resources).

[0030] Each stage statement can specify one or more processing nodes 8 to perform the operations specified in the stage statement. Each stage statement can alternatively or additionally specify one or more data structures that the corresponding stage will perform relative to it. For example, such data structures may include database / table partitions.

[0031] For example, a stage statement can instruct all processing nodes 8 to perform the operations in the stage statement. The syntax element instructing all processing nodes 8 can include "all nodes" or "all partitions," as in the stage statement 324 of the first stage of pipeline statement 302 described below, or other syntax elements. By specifying "on all partitions" for the stage, the stage statement implicitly instructs each processing node managing any one of "all partitions" to perform the stage relative to the partition it is managing. Stage statements can include syntax to explicitly specify one or more nodes that will perform the stage relative to one or more explicitly specified data structures. For example, the stage statement "stage stageC on all partitions on origin-node:" specifies that the origin-node will perform the stage's operators relative to all partitions (such as partitions of a database table). The set of nodes specified in the preceding statement can differ from the statement "stage stageC on all partitions," which does not explicitly specify any one or more nodes and therefore implicitly specifies all processing nodes managing any one of "all partitions."

[0032] As another example, a stage statement can instruct a single processing node to perform the operation in the stage statement. The syntax element used to instruct a specific single processing node 8 can be a reference or name to a specific processing node 8, or it can instruct an origin node, for example, by using the syntax element "origin" or "origin-node".

[0033] Each stage statement can have a corresponding stage identifier that can be used as a reference to the corresponding stage. Syntactic elements used to qualify a stage can include "stage[stage identifier] on [node]", where [stage identifier] can be used elsewhere in pipeline statement 20 as a reference to a stage, and [node] specifies one or more processing nodes 8 to perform the operation specified in the stage statement. That is, a first stage statement can refer to a second stage statement (and its corresponding stage) included in pipeline statement 20. For example, a first stage statement can include the stage identifier of a second stage statement, as an operation within the first stage statement, to which the output data of the first stage corresponding to the first stage statement should be sent. In other words, the second stage is the receiver of the output data generated by the first stage, and the output data generated by the first stage is the input data to the second stage. For example, in example pipeline statement 302, the stage statement of first stage statement 324 refers to "output Csv" (the stage "output Csv" in pipeline statement 302) as the stage identifier of second stage 326. In fact, second stage 326 is the next operation after first stage statement 324 in the overall distributed pipeline.

[0034] Because different stage statements within pipeline statement 20 can specify different numbers of processing nodes, pipeline statement 20 can specify a topology that includes fan-in and / or fan-out processing by processing nodes 8 between different stages. For example, a first stage statement for a first stage in pipeline statement 20 can specify "all" processing nodes 8 and refer to a second stage defined by a second stage statement in pipeline statement 20 that specifies a single processing node 8. This pipeline statement 20 specifies a topology that includes fan-in processing from the first stage to the second stage. As another example, a first stage statement for a first stage in pipeline statement 20 can specify a single processing node 8 and refer to a second stage defined by a second stage statement in pipeline statement 20 that specifies two or more processing nodes 8. This pipeline statement 20 specifies a topology that includes fan-out processing from the first stage to the second stage. When defined in this way in pipeline statement 20, other example topologies, such as one-to-one or many-to-many relationships from stage to stage, are also possible. Fan-in topologies can be useful where many devices are responsible for processing corresponding collections of data, such as IoT devices handling sensor data, distributed database servers or storage engines handling database partitioning, etc. Pipeline statement 20 specifies the fan-in topology to generate a topology that collects data from these many devices for processing at a reduced number of devices. In some cases, the collected data has already been partially processed using earlier stages specified in pipeline statement 20.

[0035] Pipeline statement 20 may also include syntactic elements indicating that pipeline statement 20 is a pipeline statement (such as "pipeline" or "pipeline statement"). Pipeline statement 20 may also include syntactic elements, or otherwise associate them with identifiers of pipeline statement 20, such as the name of pipeline statement 20. Pipeline statement 20 may include syntactic elements indicating the beginning and end of pipeline statement 20, such as "begin" and "end", other text elements, curly braces, square brackets, parentheses, comments, etc. Pipeline statement 20 may be entered into the CLI, stored in a file, executed by a script that references pipeline statement 20, and / or otherwise entered into distributed system 2. Pipeline statement may be user input. Pipeline statement 20 may include syntactic elements that enable pipeline statement 20 to be parameterized. For example, the syntax element "parameters$table$file" in a pipelined statement allows a user to invoke a stored pipelined statement 20 using the CLI or other interfaces, passing the values ​​of the $table and $file parameters. In this case, the interface can replace the parameters with the values ​​at each position in the pipelined statement template that includes these parameters, and pass the pipelined statement with the replaced values ​​as pipelined statement 20 to the processing node for processing. Alternatives for passing the values ​​of pipelined statement template parameters to the processing node are envisioned, such as using parameter vectors or environment variables.

[0036] In some examples, the formal grammar of the pipeline statement syntax can be as follows:

[0037]

[0038]

[0039] The pipeline statement specifies the name for PIPELINE. <pipelinename>Pipeline statements include the begin and end operators (BEGIN and END in this example). Stage definitions include one or more stage statements that may conform to the stage definitions in the grammar above. Other grammars are also under consideration. Similar to other example grammars described herein for specifying pipeline topologies, pipeline statements in the above grammars include specifying one or more stages, and for each stage, an ordered list of the topology of that stage (and in some cases, the next stage) and the operators of the subpipelines for that stage.

[0040] The grammar above indicates example topologies for stages. These include: ALL NODES (executes a sub-pipeline of the stage on all processing nodes); ALL PARTITIONS (executes a sub-pipeline of the stage for all partitions of the dataset); ALL PARTITIONS ON ORIGIN (executes a sub-pipeline of the stage for all partitions associated with the origin processing node); ORIGIN (executes a sub-pipeline of the stage on the origin processing node); SPECIFIC NODE node (executes a sub-pipeline of the stage on the identified node); SPECIFIC PARTITION partition (executes a sub-pipeline of the stage for the identified partition); and NODE WITH device (executes a sub-pipeline of the stage for the node with the identified device).

[0041] Example statements with the above grammar are shown below, and explained in further detail elsewhere in this specification:

[0042]

[0043]

[0044] As mentioned above, other grammars are also possible. For example, the abbreviated form of the grammar can observe and support the original elegance of the Unix pipe / filter syntax and extend it to arbitrary directed graphs of distributed parallel subpipelines. For example, the abbreviated form of the grammar described above could be:

[0045]

[0046] Example statements with the above grammar are shown below, and are shorthand restates of previous examples that conform to the preceding grammar:

[0047] scanOp LineItem|toCsvOp*|writeFileOp" / tmp / LineItem.csv"

[0048] In this example, program 9, which compiles the above abbreviated pipeline statement, checks the semantics so that the semantics are understood to be executed against all partitions (*|) and fan-in to the origin, similar to the two-stage statement of pipeline export. The abbreviated grammar also includes syntax for specifying one or more stages, for each stage, the topology of that stage (in some cases including the next stage), and an ordered list of operators for the subpipelines of that stage.

[0049] Processing node 8A can communicate with other processing nodes 8B-8N to set up distributed pipeline 12 to implement pipelined statements 20. Processing nodes 8 can collaborate to generate distributed pipeline 12. To generate distributed pipeline 12, processing nodes 8 bind not only operations between programs 9, but also operations between stages and across processing nodes 8.

[0050] To set up distributed pipeline 20, processing node 8A transmits pipeline setup specifications 22B-22N (collectively referred to as "pipeline setup specifications 22", including pipeline setup specification 22A for processing node 8A) to the corresponding processing nodes 8A-8N. In some cases, each of pipeline setup specifications 22 may be a repetition of pipeline statement 20 or a partial repetition of text in pipeline statement 20 related to the corresponding processing node 8. In some cases, each of pipeline setup specifications 22 may include configuration commands generated by processing node 8A. Configuration commands may include, for example, a list of sub-pipelines provided as a list of operations to be performed by one of the processing nodes 8 that receives any pipeline setup specification 22, and a list of one or more destination processing nodes for the output generated by executing the sub-pipelines. Each of pipeline setup specifications 22 may include the location of data to be processed. As already noted, pipeline setup specification 22A can be handled by processing node 8A to configure one or more sub-pipelines on processing node 8A for execution by processing node 8A, and at least a portion of the topology of processing node 8A involving distributed pipeline 12. In this way, processing nodes 8A can exchange information about the overall task to be performed, the location of data to be processed by operations used to complete the task, and the interconnection topology between processing nodes 8A of distributed system 2.

[0051] Each processing node 8 that obtains any one of the pipeline setup specifications 22 by receiving and / or generating any one of the pipeline setup specifications 22 can process the corresponding pipeline setup specification to configure various aspects of the distributed pipeline 12. Figure 1 In the example, processing node 8A can generate sub-pipelines of distributed pipeline 12 to bind operations executed by processing node 8A and defined by stage statements in pipeline statement 20. Similarly, processing node 8B can generate sub-pipelines of distributed pipeline 12 to bind operations to be executed by processing node 8B and defined by the same stage statements in pipeline statement 20. Similarly, processing node 8C can generate sub-pipelines of distributed pipeline 12 to bind operations executed by processing node 8C and defined by the same stage statements in pipeline statement 20, and so on. As further described below, each processing node 8 can orchestrate queues to store the inputs and outputs of operations and generate an interface between queues and operations. Processing node 8 can configure and execute multiple sub-pipelines defined by multiple different stage statements in pipeline statement 20.

[0052] As used herein, the term "subpipeline" refers to a set of one or more operations bound to a pipeline and executed by any of the processing nodes 8 to perform the operations of a single stage. A subpipeline is a subpipeline of the overall distributed pipeline defined by pipeline statement 20. A subpipeline of a stage defined by a stage statement in pipeline statement 20 may receive input data from a preceding subpipeline of a stage defined by a preceding stage statement in pipeline statement 20, or from an input interface or device such as a memory device. A subpipeline of a stage defined by a stage statement in pipeline statement 20 may send output data to a subsequent subpipeline of another stage defined by a subsequent stage statement in pipeline statement 20, or to an output interface or device such as a CLI or memory device. While a stage refers to the set of subpipelines executed by all processing nodes 8 specified in the corresponding stage statement, a subpipeline is a set of operations performed by a single processing node 8. In some cases, any single processing node 8 may execute parallel subpipelines relative to multiple data partitions or streams under the responsibility of a single processing node 8.

[0053] As part of processing any pipeline setup specification 22 and configuring the distributed pipeline 12, processing node 8 also utilizes communication channels to bind stages to each other. Communication channels can include, for example, network sockets, messaging systems or buses, remote process calls, HTTP channels, or other communication channels used to transfer data from one device 6 to another within device 6. Communication channels can also include intra-device communication "channels," such as operating system pipelines or other message buses, function calls, inter-process communication, etc. Each device 6 can be configured with network information for device 6 (e.g., IP address or network name). To set up a communication channel 13 with another device 6, device 6 can create a socket to the other device 6 using the network information. However, the setup of communication channel 13 can take various forms appropriate to the type of communication channel being set up. Communication channel 13 can be of different types between different pairs of devices 6, particularly in the heterogeneous distributed system 2.

[0054] Figure 1 The communication channels 13A-13N are included between stages, including an intra-device communication channel 13A between a first stage having a first sub-pipeline and a second stage having a second sub-pipeline, both of which are executed by processing node 8A, and an inter-device communication channel 13B-13N between a first stage having a first sub-pipeline executed by corresponding processing nodes 8B-8N and a second stage having a second sub-pipeline executed by processing node 8A.

[0055] Any of the 8 processing nodes can be configured to send the results of its local operations for a subpipeline (e.g., for the first stage) to one or more other processing nodes (in some cases, including a sending processing node) for processing by one or more subsequent subpipelines (e.g., for the second stage). Figure 1 In the example, origin processing node 8A is configured to send a send operation to send the result 24A of the first stage subpipeline to origin processing node 8A as input to the second stage subpipeline. Processing node 8C is configured to send a send operation to send the result 24B of the first stage subpipeline to origin processing node 8A as input to the second stage subpipeline. Processing node 8C is configured to send a send operation to send the result 24C of the first stage subpipeline to origin processing node 8A as input to the second stage subpipeline, and so on up to processing node 8N. The send operation can represent a TCP / IP socket operation (e.g., send()) that sends data over network 10.

[0056] Any of the 8 processing nodes can be configured to receive the results of a sub-pipeline (e.g., for a first stage) executed by other processing nodes 8 or by itself, for processing by subsequent sub-pipelines of the processing node (e.g., for a second stage). Figure 1 In the example, processing node 8A can be configured with sub-pipelines of distributed pipeline 12 for the second-stage operation. Processing node 8A can, for example, orchestrate queues to store the inputs and outputs of the operation and configure the interface between the queues and the operation. Processing node 8A can also be configured to receive operations to receive the results of the first-stage operation from each processing node 8, such as results 24A-24N (collectively referred to as "result 24"). This receive operation can be performed by processing node 8A for each communication channel 13 to obtain data sent via the communication channel. The receive operation can represent a TCP / IP socket operation (e.g., recv()) for receiving data sent via network 10. Processing node 8 will then guide the data obtained through the receive operation to the next sub-pipeline for processing.

[0057] Processing nodes 8 that share a communication channel for inter-stage communication can coordinate during communication channel setup to ensure that input to the communication channel is routed by receiving processing node 8 to the correct sub-pipeline for execution of subsequent stages. The communication channel can be associated with an identifier such as a port to associate the communication channel with a sub-pipeline. Alternatively, data transmitted via the communication channel can be tagged or otherwise associated with an identifier to associate the data with a sub-pipeline or with the overall distributed pipeline 12. For example, in the latter technique, different inter-stage bindings can use the same socket.

[0058] When processing node 8A receives result 24 from devices 6B-6N, processing node 8A can aggregate result 24, including the result 24A of the first stage operation processed locally by processing node 8A. In some examples, processing node 8A further processes the aggregated result according to a sub-pipeline for subsequent stages to generate a final result 26. Processing node 8A can export the final result 26 to user 4, for example by displaying the final result 26 on a display terminal or writing the aggregated result to a memory device. Alternatively or additionally, processing node 8A can generate a send operation to send the result to any location, such as to a dynamic query system as described in U.S. Provisional Patent Application No. 62 / 928,108, filed October 30, 2019, entitled "Dynamic Query Optimization," the entire contents of which are incorporated herein by reference.

[0059] By processing pipeline statements 20 in this manner to configure and execute distributed pipelines 12, the techniques of this disclosure can extend the traditional single-device Unix-style pipeline model to implement a distributed pipeline model, wherein the processing nodes 8 of the distributed system 2 independently set up sub-pipelines for the overall distributed pipeline 12 at least partially, and process data according to the execution topology specified in the pipeline statements 20 to execute tasks. By automatically generating distributed pipelines 12 to bind the operations to be processed according to the topology, this technique allows the user 4 to specify all operations and topologies for the distributed pipeline in the form of relatively simple pipeline statements 20, thereby avoiding the typically complex and tedious task of configuring not only each processing node 8 of the distributed system 2 with the operations to be executed, but also with the inter-node communication channel 13 of the topology between the processing nodes 8.

[0060] Figure 2A This is a block diagram illustrating an example subpipeline of a distributed pipeline according to the technology described in this disclosure. Figure 2A Device 202 can represent Figure 1 Examples of any of the six devices. Figure 2A In one example, device 202 includes a processing node 204 that runs at least one program 206 to receive and process commands from a user and / or another device.

[0061] In some cases, processing node 204 includes a storage engine that primarily performs database-related operations, and program 206 may represent a storage engine shell or other shell program. The storage engine shell may provide a high-level programming language that provides a Unix-based shell that can access the services of the operating system of device 202. Any of program 206 may also include one or more storage engine utilities.

[0062] Program 206 can receive pipeline statements or pipeline setup specifications, compile them, configure subpipelines and communication channels, distribute requests, and move large amounts of data on the grid for processing. Program 206 can acquire, process, and output data in one or more different formats, such as comma-separated values, binary, Extensible Markup Language (XML), etc.

[0063] According to the technology disclosed herein, program 206 can provide an elegant and expressive replacement for existing job specification languages ​​or task constraints that require specific instructions from the user to the machine to set up a distributed pipeline. Processing node 204 can represent Figure 1 The processing node 8 can be any one of the nodes, and program 206 can represent Figure 1 Any of the nine programs.

[0064] In some examples, program 206 may use descriptors to compile pipeline statements to generate pipeline setup specifications that can be used by processing nodes to configure states. A descriptor is a nested data structure (e.g., in JavaScript object notation or XML format) that includes information for mapping the syntax of pipeline statements to specific objects within the distributed computing system. Processing node 204 may receive pipeline statements and compile them using descriptors to generate stages, and more specifically, to generate pipeline setup specifications, in which processing nodes utilize these specifications to configure sub-pipelines to implement corresponding stages. Descriptors may specify: (1) the distributed system or mesh as a set of named (2) processing nodes specified by name and / or network address or other reachability information for inter-node communication. Each processing node includes (3) a set of one or more units, where a unit is a partition of a data structure / data subset and a set of available operators for that unit; or a specific capability (e.g., a GPU device) and a set of available operators for that unit. Program 206 can compile pipelined statements to generate and distribute pipeline setup specifications so that certain pipelined operators are executed by processing nodes with specific capabilities, such as GPUs, multi-core processors, ASICs or other discrete logic coprocessors, specific programs or other hardware or software resources.

[0065] If a stage statement within a pipeline statement instructs a sub-pipeline to execute on one or more specific nodes, such as SPECIFIC NODE 'node', then program 206 can use a descriptor to map 'node' to one of the nodes specified in the descriptor. If a sub-pipeline is to execute relative to ALL PARTITIONS, then program 206 can use a descriptor to map all cells of the specified partition to the node that includes the partition, and distribute the pipeline setup specification to such nodes. If a status statement specifies a particular operator, then program 206 can use a descriptor to map the operator to a set of operators for various cells, and then to a node that includes cells with available operators for that cell.

[0066] By using a distributed pipeline model in a potentially heterogeneous grid with a free-form structure, processing node 204 can receive pipeline statements, compile them, and cooperate with the operating system of device 202 to configure the corresponding aspects of the distributed pipeline on device 202. For example, program 206 can compile pipeline statements and automatically instantiate and configure sub-pipelines and configuration channels of the distributed pipeline by, for example, requesting operations, queues, and sockets from the operating system. In other words, program 206 orchestrates the operating system on behalf of the user to fulfill the user intent expressed in the pipeline statements. Therefore, program 206 can effectively become a high-level programming language for the user to move and process large amounts of data, for example, using a collection of potentially heterogeneous computing devices. This can be particularly advantageous when the devices are heterogeneous, as it is difficult to manually configure the devices in other ways with different operating systems, different configurations and communication mechanisms, and other differently managed methods.

[0067] In some cases, the processing node 204, operating as a storage engine, provides services to the database platform. The processing node 204 may, for example, create tables; insert rows; read rows; sort rows; aggregate rows; obtain data from external sources (such as pipelines, files, streams, etc.); compress files; perform computations; or perform other processing functions that may be associated with the storage and database platform. Example database platforms include Hadoop, Greenplum, MySQL, or any database that supports external services.

[0068] Processing node 204 can perform common database operations, such as joins, grouping by, aggregation, sorting, sorting by, filtering expressions, and other data operations, processing, and / or analysis. Processing node 204 can invoke cloud services, perform artificial intelligence or machine learning operations, custom algorithms or plugins, or standard Unix utilities such as GREP or AWK—which have compiled pipelined statements, with arguments supplied to operators by program 206, where each operator is dynamically programmable. In some cases, when executing subpipelines, program 206 dynamically instantiates operators from local or remote libraries. In this way, program 206 is not a fixed program on device 202, but rather extends the capabilities of device 202 to provide dynamic programming with relatively simple pipelined statements.

[0069] exist Figure 2A In the example, a user or another device may provide pipeline statements to program 206 running on processing node 204. Pipeline statements may specify one or more operations to be performed by device 202 in one or more stage statements, such as reading data, analyzing data, exporting data, or other operations. In this example, pipeline statements may include stage statements that specify operations 208A-208N (collectively referred to as "Operation (OP) 208") for device 202 to process data from table 210 and export the results of the operation to device 214. As a specific example, operation 208A may represent a scan operation that reads data (e.g., rows) from table 210, operation 208B may represent a search operation (e.g., grep) that searches for specific text in the rows returned by the scan operation, operation 208C may represent a sort operation that sorts the results of the search operation, and operation 208N may represent a send operation that sends the results of the operation to device 214. Each of operations 208 may represent an operation performed by a program, application, or any functional operator capable of processing data. In some cases, "operation" and "operator" can be used interchangeably in this article. Figure 2A The operations described are merely examples and may represent other types of operations.

[0070] Program 206 of processing node 204 can receive pipeline statements or pipeline setup specifications and process them to generate sub-pipelines 200 to bind operations 208. For example, program 206 orchestrates interfaces between operations 208. For example, program 206 can configure queues 212A-212N (collectively referred to as "queue 212") to pass output data from operations that are input data to subsequent operations in sub-pipelines 200. In some examples, queue 212 can queue pointers to memory buffers (e.g., memory locations storing data output by an operation for input to another operation). Queue 212 can be described herein and considered as a queue for executing the next operator of the next operation 208. In some cases, program 206 can configure input interfaces 216A-216N (collectively referred to as "input interfaces 216") for some operations 208 to obtain data and output interfaces 218A-218N (collectively referred to as "output interfaces 218") for some operations 208 to output the results of operations 208. Interfaces 216 and 218 may represent application programming interfaces (APIs) or other interfaces. Interfaces may represent or include read / write commands, queuing or IPC operations, or other operations. In some cases, message and data transfer between operators used for operations may be via another form of inter-process communication, such as via OS pipelining.

[0071] As an example, scan operation 208A uses input interface 216A to read or obtain data from table 210. Scan operation 208A uses output interface 218A to output the result of scan operation 208A (or a pointer to the memory location where the result is stored) to queue 212A. Search operation 208B uses input interface 216B to obtain data from queue 212A, such as obtaining the result of operation 208A stored in queue 212A. In some examples, search operation 208B can use input interface 216B to obtain a pointer stored in queue 212A that resolves to the memory location where the result of operation 208A is stored. Search operation 208B searches for data and stores the result of search operation 208B (or a pointer to the memory location where the result is stored) in queue 212B via output interface 218B. Sorting operation 208C uses input interface 216C to obtain the result of search operation 208B stored in queue 212B. Sorting operation 208C sorts the data obtained from queue 212B and stores the result of sorting operation 208C (or a pointer to a memory location where the result is stored) in queue 212N via output interface 218C. Export operation 208N uses input interface 216N to obtain the result of sorting operation 208C stored in queue 212N. Send operation 208N then uses output interface 218N to send the data result from sorting operation 208C to device 214, which may be, for example, a memory device, a network interface, or a user interface. In this way, the queues and interfaces enable the result of the operation to be bound to a subpipeline. In some cases, device 214 may be an example of a communication channel 13 for outputting result data from subpipeline 200 to another processing node. In this way, program 206 can configure subpipeline 200 as part of a distributed pipeline distributed across multiple nodes, where multiple stages are bound together in the topology.

[0072] Figure 2B This is a block diagram illustrating a distributed pipeline for multiple devices according to the technology described in this disclosure. In some examples, tables (e.g., Figure 2A Table 210 logically spans multiple devices and data partitions. That is, one partition of Table 210 can be stored on a device managed by device 202A, another partition of Table 210 can be stored on a device managed by device 202B, and so on.

[0073] Device 202 includes processing nodes 204A-204N (collectively referred to as "processing nodes 204"), each of which runs at least one program 206 that can cooperate to generate a distributed pipeline 220 to bind the operations processed by each device 202 according to a topology defined by pipeline statements. Processing nodes 204 can manage one or more partitions.

[0074] exist Figure 2B In the example, process node 204N's ​​program 206N can receive pipeline statements that specify that process node 204N should perform one or more operations (e.g., retrieve data, analyze data, export data, etc.) in a sub-pipeline. In this example, process node 204N is the origin, but the origin can be any of the process nodes 204 that initially received the pipeline statements from the user or agent.

[0075] Pipeline statements can include first-stage statements and second-stage statements. First-stage statements specify one or more operations to be processed by each of devices 202A-202N as part of a first stage, and second-stage statements specify one or more operations to be processed by device 202N as part of a second stage. First-stage statements can also reference second-stage statements as the destination of the results of the first stage. In this example, the first stage may include a scan operation reading from a distributed table and a search operation searching for specific text in the rows returned from the table, while the second stage of the command may include a conversion to comma-separated values ​​(CSV) operation and an export operation (e.g., exporting data to a CSV file).

[0076] The program 206N of the processing node 204N can process pipeline statements and determine, based on the syntax of the stage statements therein, the operations to be processed at different stages according to the distributed pipeline 220. In response, the device 202N generates a pipeline configuration specification and distributes the pipeline configuration specification to each of the devices 202A-202C, so that each device can configure various aspects of the distributed pipeline 220.

[0077] In this configuration, to configure various aspects of the distributed pipeline 220, each of programs 206A-206N can configure a first sub-pipeline of the distributed pipeline 220 for the first stage in the corresponding processing node 204. For example, program 206A can orchestrate queues and configure interfaces to bind operations 208A1-208AN. Program 206A can also configure a send operation (e.g., operation 208AN) that, when executed, sends the result of the local operation of the first sub-pipeline of processing node 204A to device 202N. In practice, the final operation of the sub-pipeline can be either inputting or outputting the result of the sub-pipeline to another sub-pipeline via a communication channel, or outputting the result to a device. This result can be input to a queue or other input interface for receiving the first operator of the sub-pipeline.

[0078] Programs 206B and 206C are similarly configured to process the first sub-pipelines of nodes 204B and 204C, respectively. Program 206N can be similarly configured to process the first sub-pipeline of node 204N, as follows... Figure 2B As shown. In some examples, the send operation may be bound to or otherwise utilize a communication channel to send information over the network to another processing node 204 located on a separate device or to another subpipeline within processing node 204.

[0079] Program 206N of device 202N can configure a second sub-pipeline of distributed pipeline 220. For example, program 206N configures a receive operation (e.g., operation 209N1) to receive results from devices 202A-202C, and orchestrates queues and generates interfaces to bind operations 209N1-209NK. Processing node 204N executing receive operation 209N1 receives the result data of the first stage on multiple communication channels (not shown) having multiple processing nodes 204A-204C (and in some cases, processing node 204N). Receive operation 208N1 may include aggregating the result data for processing in the second sub-pipeline of the second stage. For example, receive operation 208N1 may include an operator execution, a socket recv() call to obtain buffered socket data, and adding the buffered socket data (typically as a pointer) to a queue for the next operator for operation 209N2. In this way, the distributed pipeline 220 can bind the operations of the first stage (e.g., operations 208A1-208AN, operations 208B1-208BN, operations 208C1-208CN, and in some cases, operations 208N1-208NN (not shown)) to the operations of the second stage (e.g., operations 209N1-209NK).

[0080] Figures 3A-3C Block diagrams illustrating example implementations of distributed pipelines are shown in more detail based on one or more aspects of the disclosed technology. Figure 3A In this example, the cluster of devices implements a table of data structures (e.g., tables), such as LineItem 306, logically spanning devices 202A-202N. LineItem table 306 can represent a data structure based on Transaction Processing Performance Committee (TPC) benchmark H, a relational database table, a collection of key-value pairs from a non-relational database, a collection of one or more streams, or another data structure and / or source. In this example, table 306 is divided into one or more partitions, each managed by one of the devices 202, and more specifically, by one of the processing nodes 204. Therefore, table 306 logically spans multiple processing nodes 204 and partitions 310. In this example, device 202A manages table partitions 310A and 310B, device 202B manages table partitions 310C and 310D, and device 202N manages table partitions 310Y and 310Z. For each partition, a region of physical memory storage, such as one or more buffers, can store the data (e.g., rows) of LineItem table 306 for that partition. Figure 3A The examples are merely examples and can include any number of devices, partitions, buffers, etc. For example, any one or more devices 202 can execute multiple processing nodes, and any one or more processing nodes 204 can manage one or more partitions 310.

[0081] exist Figure 3A In the example, the program of any of the processing nodes 204 can receive a pipeline statement 302 that defines multiple stages. In this example, processing node 204A can receive pipeline statement 302, which includes a first-stage statement 324 for a corresponding first stage 334 named "scanBuffers", specifying that the corresponding first stage 334 will be executed "on all partitions", i.e., relative to all partitions 310. Pipeline statement 302 also includes a second-stage statement 326 for a corresponding second stage named "outputCsv", specifying that the corresponding second stage 336 will be executed "on origin", i.e., in this example, the processing node 204 receiving pipeline statement 302 is processing node 204A. The operation specified in the first-stage statement 324 ("scanOp LineItem|outputCsv") will be performed relative to all partitions 310 of device 202, and the operation in the second-stage statement 326 (e.g., "toCsvOp|writeFileOp' / tmp / LineItem.csv'") will be processed by the processing node 204A (e.g., the processing node that receives pipeline statement 302 and is "origin").

[0082] Processing node 204 collaborates to generate a distributed pipeline 304 by configuring sub-pipelines for stages on the processing node and binding the first stage 334 to the second stage 336 in response to the text "outputCsv" appearing as a reference to the second stage 336 as the final operation in the first stage statement 324. For example, when processing pipeline statement 302, program 206A of processing node 204A can determine that multiple stages exist in pipeline statement 302. In response, the program can distribute the pipeline setup specification generated from pipeline statement 302 to a set of collaborating programs 206 executing on devices 202A-202N.

[0083] Each of programs 206A-206N can generate a sub-pipeline of distributed pipeline 304 for the corresponding processing node 204 to be executed for the first stage 334, and program 206A of the origin processing node 204A can generate a second sub-pipeline of distributed pipeline 304 for the second stage 336.

[0084] For the first subpipeline of the first phase 323, each program 206 can orchestrate queues and generate interfaces to bind its local operations. For example, program 206A can orchestrate one or more queues to queue the result of scan operation (scanOP) 314A relative to partition 310A for input to the next operator, send 316A. Program 206A can also orchestrate one or more queues to queue the result of scan operation 314B relative to partition 310B for input to the next operator, send 316B. Program 206A can generate an input interface for scan operation 314A to read data stored in buffer 312A associated with partition 310A, and an input interface for scan operation 314B to read data stored in buffer 312B associated with partition 310B. Program 206A can generate an output interface for scan operation 314A to send the result of scan operation to one or more queues for send 316B. Program 206A can generate an output interface for scan operation 314B to send the result of the scan operation to one or more queues for send operation 316B. Programs 206B and 206N can configure similar queues, operators, and interfaces for the corresponding buffer 312, scan operation 314, and send operation 316. Data flowing through the buffer can be self-describing, non-self-describing, or otherwise (e.g., structured, semi-structured, unstructured).

[0085] Sending operations 316A-316Z (collectively referred to as "sending operations 316") send the results of scan operations 314 (which, as a whole, are the results of the first stage 334) to processing node 204A for processing the operations in the second stage 326. For example, program 206A can be configured to send sending operation 316A to send the output of scan operation 314A, and to send sending operation 316B to send the output of scan operation 314B. Program 206B can be configured to send sending operation 316C to send the output of scan operation 314C, and to send sending operation 316D to send the output of scan operation 314D. Program 206N can be configured to send sending operation 316Y to send the output of scan operation 314Y, and to send sending operation 316Z to send the output of scan operation 314Z. Sending operation 316 can be an example instance of sending operation 208N. Program 206 can be configured with an input interface for a queue through which sending operation 316 obtains results from the corresponding scan operation 314. Program 206 can configure a communication channel between the operator used for sending operation 316 and the corresponding operator used for the corresponding receiving operation 318.

[0086] For the second subpipeline of the second stage 336, the program 206A of the origin processing node 204A can be configured with a receive operation 318 to receive results from each of the devices 202B-202N. The receive operation 318 can represent multiple receive operations on multiple communication channels between the processing node 204A and each processing node 204 to receive data transmitted using the transmit operation 316.

[0087] Program 206A can also orchestrate queues and generate interfaces for operations to be performed by the origin processing node 204A. For example, program 206A can generate input interfaces and queues through which receiving operation 318 can obtain data from processing nodes 202A-202N. Program 206A can configure one or more buffers 319 to store the result data of the first stage 334 received from each of the processing nodes 204A and the output interface, through which receiving operation 318 can store the result data into buffer 319 via the output interface.

[0088] Program 206A can generate an input interface through which the CSV conversion ("toCsvOp") operation 320 can read data from buffer 319. For example, toCsvOp 320 can use the input interface to read data (e.g., binary format) stored in buffer 319. Program 206A can process toCsvOp 320 and convert the binary data to CSV format. Program 206A can output the result from toCsvOp 320 to a file storage operation ("WriteFileOP") 322 (referred to as "writeFileOp 322" or "export operation 322"). Program 206A can configure writeFileOp 322 to send CSV output interfaces or output devices to it. In this case, the output device is the file " / tmp / LineItem.csv" indicated in the second-stage statement 326.

[0089] Figure 3B This is a block diagram illustrating another example implementation of a distributed pipeline based on pipeline statement configuration according to the technology of this disclosure. Figures 3A-3C Device 202 is similar, but each diagram processes different pipeline statements, resulting in different distributed pipelines. Figure 3B In the example, pipeline statement 332 specifies that the transformation to CSV ("toCsvOp") is included in the first stage instead of the second stage, as shown below. Figure 3A As shown.

[0090] exist Figure 3B In the example, processing node 204A of device 202A receives pipeline statement 332, including a first-stage statement 344 ("stage scan scanBuffers on all partitions:scanOp LineItem|toCsvOp|outputCsv") and a second-stage statement 346 ("stage outputCsv on origin:writeFileOp" / tmp / LineItem.csv"). The operation specified in the first-stage statement 344 ("scanOp LineItem|toCsvOp|outputCsv") will be performed on all partitions 310, and the operation specified in the second-stage statement 346 ("writeFileOp' / tmp / LineItem.csv'") will be performed by the origin processing node 204A (e.g., the processing node that received pipeline statement 332).

[0091] In the first stage 354 (stage "scanBuffers") corresponding to the first stage statement 344, each processing node 204 of the management partition 310 performs the operation "scanOp LineItem|toCsvOp" on the "LineItem" table implemented using partition 310, and outputs the result to the second stage 356 (stage "outputCsv") corresponding to the second stage statement 346, which is executed by processing node 204A. The program 206 of the processing node 204 collaborates to generate a distributed pipeline 340 by configuring sub-pipelines of the first stage 354 and the second stage 356 and binding the sub-pipelines of the first stage 354 to the sub-pipelines of the second stage 356. For example, when processing pipeline statement 332, program 206A can determine that multiple stages exist in pipeline statement 332. In response, program 206A can compile pipeline statement 332 to generate a pipeline setup specification and distribute it to programs 206B-206N of devices 202B-202N.

[0092] and Figure 3A As in the example, the program 206 of the processing node 204 is configured with operators to execute specified operations, queues, input / output interfaces and / or communication channels to establish a distributed pipeline 340, which has operations and topology for the distributed system of the processing node specified by pipeline statement 332.

[0093] As mentioned above, Figure 3B The difference between pipeline statement 332 and pipeline statement 302 is that the "toCsvOp" operation is located in the first-stage statement 344 instead of the second-stage statement 346. As a result, program 206 configures distributed pipeline 340 such that toCsvOp 320 is located in the first-stage 354 instead of the second-stage 356. This improves overall processing speed and reduces the time spent by the distributed system executing pipeline statement 332 relative to the time spent by the distributed system executing pipeline statement 302, because for the toCsvOp 320 operation, multiple processing nodes 204 execute the distributed data in the distributed pipeline 340 in parallel, instead of only the origin processing node 204A with all ScanOp data results being executed in parallel, as in distributed pipeline 304.

[0094] This example demonstrates the technical advantages of the disclosed technology, allowing users to quickly experiment with and intelligently combine pipeline statements based on a high-level specification language. Reordering operators within and between stage statements can result in significant performance improvements while accomplishing the same overall task, and users can perform this reordering by simply rearranging the syntactic elements within the pipeline statements, for example, rather than manually configuring the individual subpipelines within a processing node and the execution topology between processing nodes.

[0095] Figure 3C This is a block diagram illustrating another example implementation of a distributed pipeline. The distributed pipeline 360 ​​configured by program 206 is similar to... Figure 3B The example distributed pipeline 340 is shown, but programs 206A-206N can optimize distributed pipeline 360 ​​by implementing pipeline statement 332 to avoid sending operation 316 through processing node 204A, since processing node 204A already has the results of toCsvOp 320A-320B, which can be directly output (e.g., pipeline output) to receiver 318, and is also executed by processing node 204A, i.e., collecting all results of toCsvOp 320 from all processing nodes 204.

[0096] Figure 4 This is a flowchart illustrating an example operation of a distributed system based on one or more aspects of the disclosed technology. Figure 4 It is relative to configuring and executing. Figure 3A The example distributed pipeline 304 described in the example is used.

[0097] exist Figure 4 In the example, processing node 204A receives pipeline statements 302 for a distributed pipeline 304 to be processed by multiple computing devices (402). The pipeline statements include multiple stage statements defining corresponding stages, each stage including a sub-pipeline. Processing node 204A compiles the pipeline statements to generate a pipeline setup specification 22 for processing node 204 (403). Processing node 204 processes any of the obtained pipeline setup specifications 22 to configure the sub-pipelines for the stage by instantiating and binding operators within processing node 204 for operations on the sub-pipelines (404). Furthermore, based on references to other stages within the stage statements, pipeline setup specifications 22 are generated so that processing node 204 binds different sub-pipelines for different stages to each other using a communication channel (406).

[0098] Once the distributed pipeline 304 is configured in this manner, the processing node 204 executes the distributed pipeline 304 by performing stage-by-stage sub-pipelines, including sending the result data of the sub-pipelines to other processing nodes (408) to be processed by subsequent sub-pipelines in the distributed pipeline 304. The processing node 204 outputs the result data from executing the distributed pipeline 304 to, for example, a memory, interface, or other device (410).

[0099] Figure 5 This is a block diagram of an example computing device 500 operating according to one or more techniques disclosed herein. Figure 5 A specific example of a computing device may be shown, which includes one or more processors 502 for performing... Figure 1 Equipment 6 Figures 2A-3C Device 202 or any other computing device described herein. Figure 5 In the example, computing device 500 can represent a network-enabled device, such as a server, desktop computer, laptop computer, cloud computing node, mobile phone, tablet computer, IoT device, such as a camera, sensor, television, appliance, etc. Other examples of computing device 500 can be used for other instances. Although for illustrative purposes... Figure 5 The image shows 500 as a standalone computing device, but a computing device can be any component or system including one or more processors or other suitable computing environments for executing software instructions, and does not necessarily need to include, for example, a... Figure 5 One or more elements shown (e.g., communication unit 506; and in some examples, components such as memory device 508 may not be in computing device 500).

[0100] like Figure 5 As illustrated in the specific examples, computing device 500 includes one or more processors 502, one or more input devices 504, one or more communication units 506, one or more output devices 512, one or more memory devices 508, and one or more user interface (UI) devices 510. In one example, computing device 500 also includes one or more application programs 522 and an operating system 516 executable by computing device 500. Each of components 502, 504, 506, 508, 510, and 512 is coupled (physically, communicatively, and / or operatively) for inter-component communication. In some examples, communication channel 514 may include a system bus, network connection, inter-process communication data structure, or any other method for communicating data. As an example, components 502, 504, 506, 508, 510, and 512 may be coupled by one or more communication channels 514.

[0101] In one example, processor 502 is configured to implement functional and / or process instructions for execution within computing device 500. For example, processor 502 may be able to process instructions stored in memory device 508. Examples of processor 502 may include any one or more of a microprocessor, controller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry.

[0102] One or more memory devices 508 may be configured to store information within computing device 500 during operation. In some examples, memory device 508 is described as a computer-readable storage medium. In some examples, memory device 508 is temporary memory, meaning that the primary purpose of memory device 508 is not long-term storage. In some examples, memory device 508 is described as volatile memory, meaning that memory device 508 does not maintain its stored contents when the computer is turned off. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, memory device 508 is used to store program instructions executed by processor 502. In one example, memory device 508 is used by software or an application running on computing device 500 to temporarily store information during program execution.

[0103] In some examples, memory device 508 also includes one or more computer-readable storage media. Memory device 508 can be configured to store a larger amount of information than volatile memory. Memory device 508 can also be configured for long-term storage of information. In some examples, memory device 508 includes non-volatile storage elements. Examples of such non-volatile storage elements include magnetic hard disks, optical disks, floppy disks, flash memory, or electronically programmable memory (EPROM) or electronically erasable and programmable (EEPROM) memory.

[0104] In some examples, computing device 500 also includes one or more communication units 506. In one example, computing device 500 uses communication unit 506 to communicate with external devices via one or more networks (e.g., one or more wired / wireless / mobile networks). Communication unit 506 may include a network interface card, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device capable of sending and receiving information. Other examples of such network interfaces may include 3G, 4G, and Wi-Fi radios. In some examples, computing device 500 uses communication unit 506 to communicate with external devices.

[0105] In one example, computing device 500 also includes one or more user interface devices 510. In some examples, user interface device 510 is configured to receive input from a user via haptic, audio, or video feedback. Examples of user interface device 510 include presence-sensitive displays, mice, keyboards, voice response systems, cameras, microphones, or any other type of device for detecting commands from the user. In some examples, presence-sensitive displays include touch-sensitive screens.

[0106] One or more output devices 512 may also be included in the computing device 500. In some examples, the output device 512 is configured to provide output to a user using tactile, audio, or video stimuli. In one example, the output device 512 includes a presence-sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting signals into a suitable form that is understandable to humans or machines. Additional examples of the output device 512 include a speaker, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD), or any other type of device capable of generating understandable output to a user.

[0107] The computing device 500 may include an operating system 516. In some examples, the operating system 516 controls the operation of components of the computing device 500. For example, in one example, the operating system 516 facilitates communication between one or more applications 522 and processor 502, communication unit 506, memory device 508, input device 504, user interface device 510, and output device 512. The application 522 may also include program instructions and / or data executable by the computing device 500.

[0108] Processing node 524 may include instructions for causing computing device 500 to run one or more programs 526 to perform the techniques described in this disclosure. Program 526 may represent... Figure 1 Program 9 or Figure 2- Figure 3C Example instances of any of the procedures 206. For example, computing device 500 may receive pipeline statements via one or more user interface devices 510 or one or more input devices 504 and send pipeline statements for processing by procedure 526 of processing node 524. Procedure 526 may include instructions that enable computing device 500 to assist in configuring distributed pipelines on a distributed system according to the techniques of this disclosure. For example, procedure 526 may interpret various syntactic elements of pipeline statements to generate a distributed pipeline that binds operations for sub-pipelines within processing nodes of a stage, and sometimes binds sub-pipelines for different stages together via communication channels between processing nodes. For example, procedure 526 may interpret the syntax of pipeline statements and determine that the pipeline statements include a first stage and a second stage, in which all processing nodes perform one or more operations, and in the second stage, the original node performs one or more different operations. Procedure 526 may generate a pipeline setup specification based on the pipeline statements and distribute the pipeline setup specification to other processing nodes via communication unit 506.

[0109] Program 526 can generate a pipeline setup specification for processing node 524. In some cases, computing device 500 receives a pipeline setup specification generated by another processing node via communication unit 506 or an input device. In any case, program 526 processes the pipeline setup specification to instantiate operators and binds the operators together using input / output channels to generate sub-pipelines of a distributed pipeline, and in some cases binds the sub-pipelines to another sub-pipeline executed by another computing device.

[0110] As an example, program 526 can issue low-level calls to operating system 516 to allocate memory (e.g., queues) for local operations in the subpipeline. Program 526 can also configure the interface between the operations and the queues. Program 526 can also generate low-level calls to the kernel to configure a receive operator to receive the results of operations processed by other devices.

[0111] Once any specified subpipeline and communication channel are configured, program 526 can execute the subpipeline by executing operators for the corresponding operations of the subpipeline. In this way, computing device 500 operates as part of a distributed system to configure and execute distributed pipelines.

[0112] The techniques described in this disclosure can be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, aspects of the described techniques can be implemented within one or more processors, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuits, and any combination of these components. The terms "processor" or "processing circuitry" generally refer to any of the aforementioned logic circuits, alone or in combination with other logic circuits, or any other equivalent circuitry. Control units, including hardware, can also perform one or more of the techniques of this disclosure.

[0113] Such hardware, software, and firmware can be implemented within the same device or in separate devices to support the various operations and functions described in this disclosure. Furthermore, any of the described units, modules, or components can be implemented together or separately as discrete but interoperable logical devices. Describing different features as modules, units, or engines is intended to emphasize different functional aspects and does not necessarily imply that such modules or units must be implemented by separate hardware or software components. Rather, the functionality associated with one or more modules or units can be performed by separate hardware or software components, or integrated within common or separate hardware or software components.

[0114] The techniques described in this disclosure can also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium containing instructions. Instructions embedded or encoded in a computer-readable storage medium can cause a programmable processor or other processor to perform the method, for example, when the instructions are executed. Computer-readable storage media may include random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory, hard disk, CD-ROM, floppy disk, magnetic tape, magnetic media, optical media, or other computer-readable media.< / pipelinename>

Claims

1. A distributed computing system for data processing, comprising: Multiple computing devices are configured to obtain pipeline statements, the pipeline statements including a first-stage statement that defines a first stage to include a first sub-pipeline and designates a second stage as the recipient of the result data of the first sub-pipeline, the pipeline statements also including a second-stage statement that defines a second stage to include a second sub-pipeline. The plurality of computing devices include a first processing node, which is configured to configure a first sub-pipeline in the first processing node based at least on a first pipeline setup specification generated by processing the pipeline statements, to send the result data of the first sub-pipeline in the first processing node to a second processing node. The plurality of computing devices include the second processing node, which is configured to configure the second sub-pipeline in the second processing node based at least on the second pipeline setup specification generated by processing the pipeline statements, and input the result data of the first sub-pipeline in the first processing node into the second sub-pipeline in the second processing node.

2. The distributed computing system according to claim 1, wherein The first subpipeline includes one or more first operations, and The second sub-pipeline includes one or more second operations.

3. The distributed computing system according to claim 1 further includes: A compiler computing device is configured to process the pipelined statements to generate the first pipeline setup specification and the second pipeline setup specification.

4. The distributed computing system of claim 3, wherein, The compiler computing device includes the second processing node.

5. The distributed computing system of claim 1, wherein, The pipeline statements include human-readable text.

6. The distributed computing system according to any one of claims 1-5, wherein The second phase statement includes a phase identifier for the second phase, and Specifically, in order to designate the second stage as the recipient of the result data of the first sub-pipeline, the first stage statement includes the stage identifier of the second stage as the operation of the first sub-pipeline, and The distributed computing system is configured to generate a first pipeline setup specification based on the stage identifier of the second stage included in the first stage statement, so as to configure the first sub-pipeline in the first processing node to send the result data of the first sub-pipeline in the first processing node to the second processing node.

7. The distributed computing system according to any one of claims 1, 2, or 5, wherein The plurality of computing devices includes a compiler computing device configured to obtain the pipelined statements and process the pipelined statements to generate the first pipeline setup specification and the second pipeline setup specification.

8. The distributed computing system of claim 7, wherein, The compiler computing device includes the second processing node.

9. The distributed computing system according to any one of claims 1-5, in, The first subpipeline includes one or more first operations, and In order to configure the first subpipeline, the first processing node is configured to instantiate a corresponding operator for the one or more first operations, and bind at least one pair of the operators using input / output channels.

10. The distributed computing system according to any one of claims 1-5, in, The first subpipeline includes one or more first operations. Specifically, to configure the first sub-pipeline, the first processing node is configured to instantiate corresponding operators for the one or more first operations, and The final operator of the operator includes a send operator, which is used to send the result data of the first sub-pipeline to the second processing node via a communication channel.

11. The distributed computing system according to any one of claims 1-5, further comprising: The third processing node is configured to configure the first sub-pipeline in the third processing node, based at least on the third pipeline setup specification generated from the pipeline statement, to send the result data of the first sub-pipeline in the third processing node to the second processing node.

12. The distributed computing system according to any one of claims 1-5, in, The second processing node is configured to configure the first sub-pipeline in the second processing node based at least on the second pipeline setup specification generated from the pipeline statement, so as to provide the result data of the first sub-pipeline in the second processing node as input to the second sub-pipeline in the second processing node.

13. The distributed computing system according to any one of claims 1-5, in, The first stage statement defines the first stage to specify the execution of the first one or more processing nodes of the first subpipeline, the first one or more processing nodes including the first processing node, and The distributed computing system is configured to generate a first pipeline setup specification for the first processing node based on the first stage statement, so as to configure the first sub-pipeline in the first processing node.

14. The distributed computing system according to claim 13, in, The first stage statement includes one or more syntax elements that specify the first one or more processing nodes to execute the first subpipeline, and The distributed computing system is configured to generate a pipeline setup specification for configuring each of the first one or more processing nodes based on one or more of the syntax elements.

15. The distributed computing system according to claim 13, in, The second phase statement defines the second phase to specify a second or more processing nodes that execute the second subpipeline, the second or more processing nodes including the second processing node, and The distributed computing system is configured to generate a second pipeline setup specification for the second processing node based on the second phase statement, so as to configure the second sub-pipeline in the second processing node.

16. The distributed computing system according to any one of claims 1-5, wherein, The first stage statement defines the first stage to specify one or more first partitions of the data structure, and the first subpipeline is to be executed against the first partition.

17. The distributed computing system according to any one of claims 1-5, wherein, Each of the first processing node and the second processing node includes one of a bare metal server, a virtual machine, a container, a shell program, or a storage engine.

18. The distributed computing system according to any one of claims 1-5, in, The first processing node is configured to execute the first sub-pipeline within the first processing node to generate first result data, and to send the first result data to the second processing node via a communication channel. The second processing node is configured to execute the second sub-pipeline in the second processing node to process the first result data to generate the second result data and output the second result data.

19. A method for data processing, comprising: A command comprising multiple stages is received by one of a plurality of devices connected via a network, wherein each stage includes one or more operations to be processed by one or more of the plurality of devices, and wherein the one or more operations in the stage will be processed in different topologies; The command is distributed from the device to other devices among the plurality of devices, such that each of the other devices generates a first sub-pipeline of the distributed pipeline, wherein the first sub-pipeline is bound to one or more operations processed in a first topology of the different topologies; and The device generates a second sub-pipeline of the distributed pipeline, wherein the second sub-pipeline is bound to one or more operations processed in a second topology in a different topology.

20. A method for data processing, comprising: A pipeline statement is obtained by a distributed computing system including a first processing node and a second processing node. The pipeline statement includes a first stage statement that defines the first stage to include a first sub-pipeline and designates a second stage as the recipient of the result data of the first sub-pipeline. The pipeline statement also includes a second stage statement that defines the second stage to include a second sub-pipeline. Based at least on the first pipeline setup specification generated from the pipeline statement, configure the first sub-pipeline in the first processing node to send the result data of the first sub-pipeline in the first processing node to the second processing node; as well as Based at least on the second pipeline setup specification generated from the pipeline statement, the second sub-pipeline in the second processing node is configured to input the result data of the first sub-pipeline in the first processing node into the second sub-pipeline in the second processing node.

21. A computer-readable storage medium encoded with instructions that cause one or more programmable processors to implement a distributed computing system according to any one of claims 1-18.

Images