A data distribution method and apparatus for distributed graph computation

Abstract

A data distribution method for distributed graph computation, computing nodes in a distributed system support performing a distributed graph computation task in multiple computation modes; the distributed graph computation task includes a first subtask as an upstream task and a second subtask as a downstream task; the first subtask and the second subtask are concurrently executed by multiple computing nodes; the method comprises: receiving a data acquisition request sent by any target downstream computing node in multiple downstream computing nodes for executing the second subtask; the data acquisition request includes a data acquisition condition determined by the target downstream computing node based on a computation mode adopted for executing the second subtask; different computation modes adopted by the target downstream computing node for executing the second subtask correspond to different data acquisition conditions respectively; target data satisfying the data acquisition condition is acquired from data generated by executing the first subtask; and the target data is distributed to the target downstream computing node.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of graph computing, and in particular, relate to a data distribution method and device for distributed graph computing. Background Art

[0002] To adapt to diverse business needs, more and more graph computing engines have begun to support executing graph computing tasks according to a variety of different computing modes. For example, graph computing engines can support executing graph computing tasks according to stream processing mode, batch processing mode, and graph processing mode.

[0003] In distributed graph computing scenarios, regardless of the graph computing engine's execution mode, data shuffling is inevitable between upstream and downstream tasks involved in the distributed graph computation. Data shuffling refers to the process by which upstream tasks in a distributed graph computation distribute data to downstream tasks over the network.

[0004] However, in actual applications, under different computing modes, the shuffle methods used between upstream and downstream tasks often differ. This requires the graph computing engine to provide different data shuffle mechanisms for different computing modes, which may increase the system complexity of the graph computing engine and cause excessive system overhead. Summary of the Invention

[0005] This specification proposes a data distribution method for distributed graph computing, wherein computing nodes in a distributed system support executing a distributed graph computing task according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; wherein the first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system; the method is applied to any target upstream computing node among multiple upstream computing nodes executing the first subtask, and the method includes:

[0006] receiving a data acquisition request sent by any target downstream computing node of the plurality of downstream computing nodes for executing the second subtask; wherein the data acquisition request contains a data acquisition condition determined by the target downstream computing node based on a computing mode adopted by the target downstream computing node for executing the second subtask; the data acquisition condition is used to acquire input data required by the target downstream computing node for executing the second subtask from data generated by the target upstream computing node for executing the first subtask; different computing modes adopted by the target downstream computing node for executing the second subtask correspond to different data acquisition conditions respectively;

[0007] in response to the data acquisition request, acquiring target data satisfying the data acquisition condition from the data generated by executing the first subtask;

[0008] downloading the target data to the target downstream computing node, so as to continue executing the second subtask by the target downstream computing node taking the target data as input data.

[0009] Optionally, the data generated by executing the first subtask contains data shards respectively allocated to the plurality of downstream computing nodes for executing the second subtask; each data shard contains at least one batch of data; and each batch of data contains at least one data block.

[0010] wherein the number of batches of data contained in the data shard allocated to any downstream computing node is the number corresponding to the computing mode adopted by the downstream computing node for executing the second subtask.

[0011] Optionally, the data acquisition condition includes a shard identifier of the data shard allocated to the target downstream computing node by the target upstream computing node, a batch identifier of a starting batch of data to be acquired, and a batch number of data batches to be acquired.

[0012] in response to the data acquisition request, acquiring target data satisfying the data acquisition condition from the data generated by executing the first subtask, comprising:

[0013] in response to the data acquisition request, determining a target data shard corresponding to the shard identifier from the data shards respectively allocated to the plurality of downstream computing nodes;

[0014] determining a target data batch corresponding to the batch identifier from at least one batch of data contained in the target data shard, and taking the target data batch as a starting data batch; and sequentially acquiring at least one batch of data corresponding to the value of the batch number as the target data.

[0015] Optionally, the multiple computing modes include stream processing mode, batch processing mode and iterative graph computing mode;

[0016] If the computing mode adopted by the target downstream computing node to execute the second subtask is stream processing mode, the target data shard contains multiple batches of data, the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data, and the value of the batch number is infinite;

[0017] If the computing mode adopted by the target downstream computing node to execute the second subtask is batch mode, the target data shard contains only one batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the only batch of data, and the value of the batch quantity is 1;

[0018] The batch identifiers of the multiple batches of data are monotonically increasing.

[0019] Optionally, the iterative graph computing mode includes an iterative graph computing mode executed in a stream processing manner; and an iterative graph computing mode executed in a batch processing manner;

[0020] If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a stream processing manner, the first subtask is a computing task of any one round of iteration in the iterative graph computing, and the second subtask is a computing task of the next round of iteration of the any one round of iteration; the target data shard contains multiple batches of data; the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data; and the value of the batch quantity is infinite;

[0021] If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a batch processing manner, the first subtask is a computing task of any iterative round in the iterative graph computing, and the second subtask is a computing task of the next iterative round of the iterative round; the target data shard contains only one batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the only batch of data, and the value of the batch quantity is 1;

[0022] The batch identifiers of the multiple batches of data are monotonically increasing.

[0023] Optionally, the data acquisition condition includes a shard identifier of a data shard allocated by the target upstream computing node to the target downstream computing node, and a maximum batch identifier of all data batches to be acquired;

[0024] In response to the data acquisition request, acquiring target data that meets the data acquisition condition from the data generated by executing the first subtask includes:

[0025] In response to the data acquisition request, determining a target data shard corresponding to the shard identifier from the data shards respectively allocated to the plurality of downstream computing nodes;

[0026] At least one batch of data whose batch identifier is not greater than the maximum batch identifier is obtained from at least one batch of data included in the target data shard as the target data.

[0027] Optionally, the multiple computing modes include stream processing mode, batch processing mode and iterative graph computing mode;

[0028] If the computing mode adopted by the target downstream computing node to execute the second subtask is stream processing mode, the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinite;

[0029] If the computing mode adopted by the target downstream computing node to execute the second subtask is batch processing mode, and the target data shard contains only one batch of data, the value of the maximum batch identifier is the batch identifier corresponding to the only batch of data;

[0030] The batch identifiers of the multiple batches of data are monotonically increasing.

[0031] Optionally, the iterative graph computing mode includes an iterative graph computing mode executed in a stream processing manner; and an iterative graph computing mode executed in a batch processing manner;

[0032] If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a stream processing manner, the first subtask is a computing task of any iterative round in the iterative graph computing, and the second subtask is a computing task of the next iterative round of any iterative round; the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinite;

[0033] If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a batch processing manner, the first subtask is a computing task of any iterative round in the iterative graph computing, and the second subtask is a computing task of the next iterative round of the iterative round; the target data shard contains only one batch of data, and the value of the maximum batch identifier is the batch identifier corresponding to the only batch of data;

[0034] The batch identifiers of the multiple batches of data are monotonically increasing.

[0035] Optionally, the data acquisition request is an asynchronous request;

[0036] In response to the data acquisition request, acquiring target data that meets the data acquisition condition from the data generated by executing the first subtask includes:

[0037] Asynchronously responding to the data acquisition request, acquiring target data that meets the data acquisition condition from the data generated by executing the first subtask.

[0038] Optionally, the target data includes at least one batch of data obtained from a data shard allocated to the target downstream computing node;

[0039] Sending the target data to the target downstream computing node includes:

[0040] The data blocks contained in the acquired at least one batch of data are sent to the target downstream computing node in a streaming manner.

[0041] Optionally, multiple downstream computing nodes for executing the second subtask use different computing modes to execute the second subtask.

[0042] This specification also provides a data distribution method for distributed graph computing, wherein computing nodes in a distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; wherein the first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system; the method is applied to any target downstream computing node among multiple downstream computing nodes executing the second subtask, and the method includes:

[0043] Determining a data acquisition condition based on a computing mode used to execute the second subtask; wherein the data acquisition condition is used to obtain input data required for executing the second subtask from any target upstream computing node among multiple upstream computing nodes executing the first subtask;

[0044] generating a data acquisition request including the data acquisition condition, and sending the data acquisition request to the target upstream computing node, so that the target upstream computing node acquires target data that meets the data acquisition condition from data generated by executing the first subtask;

[0045] Receive the target data sent by the target upstream computing node, and use the target data as input data to continue executing the second subtask.

[0046] Optionally, the data generated by executing the first subtask includes data shards allocated to multiple downstream computing nodes that execute the second subtask; wherein, the data shards include at least one batch of data; any batch of data includes at least one data block; the number of batches of data included in the data shards allocated to any downstream computing node is the number corresponding to the computing mode adopted by the downstream computing node to execute the second subtask.

[0047] Optionally, the multiple computing modes include stream processing mode, batch processing mode and iterative graph computing mode;

[0048] The data acquisition condition includes a shard identifier of a data shard allocated by the target upstream computing node to the target downstream computing node, a batch identifier of a starting data batch to be acquired, and a batch quantity of data batches to be acquired;

[0049] If the computing mode adopted by the target downstream computing node to execute the second subtask is stream processing mode, the target data shard contains multiple batches of data, the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data, and the value of the batch number is infinite;

[0050] If the computing mode adopted by the target downstream computing node to execute the second subtask is batch mode, the target data shard contains only one batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the only batch of data, and the value of the batch quantity is 1;

[0051] The batch identifiers of the multiple batches of data are monotonically increasing.

[0052] Optionally, the iterative graph computing mode includes an iterative graph computing mode executed in a stream processing manner; and an iterative graph computing mode executed in a batch processing manner;

[0053] If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a stream processing manner, the first subtask is a computing task of any one round of iteration in the iterative graph computing, and the second subtask is a computing task of the next round of iteration of the any one round of iteration; the target data shard contains multiple batches of data; the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data; and the value of the batch quantity is infinite;

[0054] If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a batch processing manner, the first subtask is the computing task of any round of iteration in the iterative graph computing, and the second subtask is the computing task of the next round of iteration of the any round of iteration; the target data shard contains only one batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the only batch of data, and the value of the batch quantity is 1.

[0055] The batch identifiers of the multiple batches of data are monotonically increasing.

[0056] Optionally, the multiple computing modes include stream processing mode, batch processing mode and iterative graph computing mode;

[0057] The data acquisition condition includes a shard identifier of a data shard allocated by the target upstream computing node to the target downstream computing node, and a maximum batch identifier of all data batches to be acquired;

[0058] If the computing mode adopted by the target downstream computing node to execute the second subtask is stream processing mode, the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinite;

[0059] If the computing mode adopted by the target downstream computing node to execute the second subtask is batch processing mode, and the target data shard contains only one batch of data, the value of the maximum batch identifier is the batch identifier corresponding to the only batch of data;

[0060] The batch identifiers of the multiple batches of data are monotonically increasing.

[0061] Optionally, the iterative graph computing mode includes an iterative graph computing mode executed in a stream processing manner; and an iterative graph computing mode executed in a batch processing manner;

[0062] If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a stream processing manner, the first subtask is a computing task of any iterative round in the iterative graph computing, and the second subtask is a computing task of the next iterative round of any iterative round; the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinite;

[0063] If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a batch processing manner, the first subtask is a computing task of any iterative round in the iterative graph computing, and the second subtask is a computing task of the next iterative round of the iterative round; the target data shard contains only one batch of data, and the value of the maximum batch identifier is the batch identifier corresponding to the only batch of data;

[0064] The batch identifiers of the multiple batches of data are monotonically increasing.

[0065] Optionally, the data acquisition request is an asynchronous request.

[0066] Optionally, the target data includes at least one batch of data obtained from a data shard allocated by the target upstream computing node to the target downstream computing node;

[0067] Receiving the target data sent by the target upstream computing node includes:

[0068] Receive data blocks included in the at least one batch of data sent by the target upstream computing node in a streaming manner.

[0069] Optionally, multiple downstream computing nodes for executing the second subtask use different computing modes to execute the second subtask.

[0070] In the above embodiment, a unified data distribution method can be provided for the various computing modes supported by the distributed system that performs distributed graph computing. There is no need to provide different data distribution methods for different computing modes, which can reduce the complexity of system development and maintenance and allow the system to adapt to different computing modes more flexibly. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] Figure 1 This is a schematic diagram illustrating data distribution from an upstream task to a downstream task under different computing modes in an embodiment of this specification;

[0073] Figure 2 This is a flow chart of a data distribution method for distributed graph computing shown in an embodiment of this specification;

[0074] Figure 3 is a schematic diagram of an upstream task and a downstream task shown in an embodiment of the present specification;

[0075] Figure 4 is a schematic diagram of concurrent execution of an upstream task and a downstream task shown in an embodiment of the present specification;

[0076] Figure 5 is a schematic diagram of concurrent execution of an upstream task and a downstream task shown in an embodiment of the present specification;

[0077] Figure 6 is a data structure diagram of data generated by an upstream computing node executing a first subtask shown in an embodiment of the present specification;

[0078] Figure 7 is a flowchart of another data distribution method for distributed graph computing shown in an embodiment of the present specification;

[0079] Figure 8 is a schematic structural diagram of an electronic device shown in an embodiment of the present specification;

[0080] Figure 9 is a block diagram of a data distribution apparatus for distributed graph computing shown in an embodiment of the present specification;

[0081] Figure 10 is a block diagram of a data distribution apparatus for distributed graph computing shown in an embodiment of the present specification. DETAILED DESCRIPTION

[0082] In order to enable persons skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be described clearly and completely in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, not all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by persons skilled in the art without creative labor should belong to the scope of protection of the present specification.

[0083] In the distributed graph computing scenario, the graph computing engine supporting multiple computing modes usually adopts different data distribution methods if data distribution (shuffle) is needed between the upstream tasks and the downstream tasks contained in the executed distributed graph computing in different computing modes.

[0084] Please refer to Figure 1 , Figure 1 is a schematic diagram of data distribution from an upstream task to a downstream task in different computing modes shown in the present specification.

[0085] like Figure 1 As shown in the figure, if the graph computing engine uses batch processing mode, because batch processing processes large amounts of data and generally has lower requirements for data latency, in this computing mode, after the upstream task completes execution, it can usually store the generated batch data in local or external storage. When the downstream task starts, it can send a request to the upstream task to actively pull batch data from this local or external storage as input data. This data distribution method is often called pull-based shuffle, which means that the downstream task actively pulls the required large batches of data from the upstream task.

[0086] like Figure 1 As shown in the figure, if the graph computing engine uses stream processing, which has high requirements for data latency and processes smaller amounts of data, after the upstream task completes, the generated data can usually be divided into several small batches, and then actively pushed to the downstream task in batches. This data distribution method is often called push-based shuffle, which means that the upstream task actively pushes small batches of data to the downstream task.

[0087] like Figure 1 As shown in the figure, if the graph computing engine uses the graph computing mode, due to its relatively special nature, after completing one round of graph computing, the compute nodes will continue to send the generated data to themselves as input for the next round of graph computing. Based on this characteristic, in the graph computing mode, data distribution can be performed using either the pull-based shuffle method or the push-based shuffle method.

[0088] For example, see Figure 1 For graph computing scenarios with a small amount of data to process, iterative graph computing can be performed using stream processing, and the data generated by the previous iterative computing can be distributed to the next iterative computing as input data using the Push-based Shuffle method. For graph computing scenarios with a large amount of data to process, iterative graph computing can be performed using batch processing, and the data generated by the previous iterative computing can be distributed to the next iterative computing as input data using the Pull-based Shuffle method.

[0089] Under different computing modes, the shuffle methods used by upstream and downstream tasks when shuffling data are usually different. This requires the graph computing engine to provide different data shuffle mechanisms for different computing modes.

[0090] As can be seen, while distributed systems support multiple computing modes when executing distributed graph computations, upstream and downstream tasks typically use different data distribution methods. This requires the graph computing engine to provide different data shuffle mechanisms for different computing modes, which can increase system complexity and overhead. Furthermore, it can also make it difficult for the graph computing engine to flexibly adapt to different computing modes.

[0091] Based on this, this specification proposes a technical solution that provides a unified data distribution method for multiple computing modes supported by a distributed system that performs distributed graph computing.

[0092] See Figure 2 , Figure 2 This specification provides a flow chart of a data distribution method for distributed graph computing. The computing nodes in the distributed system support executing distributed graph computing tasks according to multiple computing modes. The distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task. The first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system. The method is applied to any target upstream computing node among multiple upstream computing nodes executing the first subtask. The method includes the following execution process:

[0093] Step 202: Receive a data acquisition request sent by any target downstream computing node among multiple downstream computing nodes for executing the second subtask; wherein the data acquisition request includes a data acquisition condition determined by the target downstream computing node based on the computing mode adopted for executing the second subtask; the data acquisition condition is used to obtain input data required for the target downstream computing node to execute the second subtask from data generated by the target upstream computing node when executing the first subtask; different computing modes adopted by the target downstream computing node to execute the second subtask correspond to different data acquisition conditions;

[0094] The computing nodes in the above distributed system can specifically support the execution of distributed graph computing tasks according to multiple computing modes.

[0095] For example, in practical applications, a graph computing engine supporting performing distributed graph computing tasks in multiple computing modes can be deployed on the computing nodes in the distributed system, so that the computing nodes in the distributed system can also have the ability to perform distributed graph computing tasks in multiple computing modes.

[0096] It should be noted that, in practical applications, the computing nodes in the distributed system can be a separate node device in the computing cluster, or a thread running on the node device for executing a graph computing task.

[0097] The distributed graph computing task can include a first subtask as an upstream task and a second subtask as a downstream task.

[0098] In practical applications, the distributed graph computing task usually includes multiple subtasks with data dependency. In these subtasks, if the data generated after the execution of a target subtask (i.e., the output data of the target subtask) is used as the input data of another subtask, the target subtask can be an upstream task, and the other subtask can be a downstream task corresponding to the upstream task.

[0099] It should be noted that the specific types of the subtasks as upstream tasks and downstream tasks in the distributed graph computing task are not limited in the present specification. In practical applications, the types of the subtasks as upstream tasks and downstream tasks can be the same or different.

[0100] For example, referring to Figure 3 In one example, the subtask as an upstream task can be a map task included in a graph computing task, and the subtask as a downstream task can be a sink task included in the graph computing task. In another example, the subtask as an upstream task can be a source task included in a graph computing task, and the subtask as a downstream task can be an iterative graph task included in the graph computing task. Alternatively, the subtask as an upstream task can be an iterative graph task included in a graph computing task, and the subtask as a downstream task can be a sink task included in the graph computing task.

[0101] In some embodiments, in the application scenario of distributed graph computing, the first subtask and the second subtask can be concurrently executed by multiple computing nodes in the distributed system in order to fully utilize the computing resources in the distributed system. That is, whether it is an upstream task or a downstream task, it can be a task with a certain degree of concurrency.

[0102] It should be noted that, in actual applications, the concurrency of upstream tasks and downstream tasks may be the same or different.

[0103] For example, see Figure 4 , Figure 4 This is a schematic diagram of concurrently executing upstream tasks and downstream tasks shown in this specification.

[0104] like Figure 4 As shown, taking the example that the upstream task and the downstream task can have the same concurrency, the upstream task can be concurrently executed by a total of three upstream computing nodes, nodes 1-3 in the distributed system, and the downstream task can also be concurrently executed by a total of three downstream computing nodes, nodes AC in the distributed system.

[0105] In some embodiments, the data generated by the execution of the first subtask by multiple upstream computing nodes for concurrently executing the above-mentioned first subtask will be distributed as input data to the multiple downstream computing nodes for concurrently executing the above-mentioned second subtask, and the multiple downstream computing nodes will continue to execute the above-mentioned second subtask in parallel based on the data distributed by the upstream computing nodes.

[0106] Among them, in the distributed graph computing scenario, for any upstream computing node among the above-mentioned multiple upstream computing nodes, the data generated by executing the first subtask can be partitioned according to the concurrency of the downstream computing node in executing the above-mentioned second subtask, so as to further divide the data generated by executing the first subtask into multiple data shards corresponding one-to-one to the multiple downstream computing nodes executing the above-mentioned second subtask, and then the multiple data shards are respectively allocated to each downstream computing node among the above-mentioned multiple downstream computing nodes.

[0107] For example, see Figure 5 , Figure 5 This is another schematic diagram of concurrently executing upstream tasks and downstream tasks shown in this specification.

[0108] like Figure 5As shown, taking the example that the upstream task is concurrently executed by three upstream computing nodes, nodes 1-3 in the distributed system, and the downstream task is concurrently executed by three downstream computing nodes, nodes AC in the distributed system, the upstream node 1 can divide the data generated by executing the first subtask into slice1-A, slice1-B and slice1-C according to the concurrency 3 of the downstream computing nodes, and distribute slice1-A to the downstream node A, slice1-B to the downstream node B, and slice1-C to the downstream node C. Similarly, the upstream node 2 can also divide the data generated by executing the first subtask into slice2-A, slice2-B and slice2-C according to the concurrency 3 of the downstream computing node, and distribute slice2-A to the downstream node A, slice2-B to the downstream node B, and slice2-C to the downstream node C; the upstream node 2 can also divide the data generated by executing the first subtask into slice3-A, slice3-B and slice3-C according to the concurrency 3 of the downstream computing node, and distribute slice3-A to the downstream node A, slice3-B to the downstream node B, and slice3-C to the downstream node C. Finally, the input data of the downstream node A when executing the second subtask is slice1-A, slice2-A and slice3-A, the input data of the downstream node B when executing the second subtask is slice1-B, slice2-B and slice3-B, and the input data of the downstream node C when executing the second subtask is slice1-C, slice2-C and slice3-C.

[0109] In some embodiments, each upstream computing node that executes the first subtask may, in addition to dividing the output data generated by executing the first subtask into multiple data shards corresponding one-to-one to multiple downstream computing nodes that execute the second subtask according to the concurrency of the downstream computing nodes, also use a unified data structure to abstract each data shard into the form of a data stream.

[0110] It should be noted that, in one case, the upstream computing node may, after completing the execution of the above-mentioned first subtask, use a unified data structure to further abstract each data slice divided from the output data generated by executing the first subtask into the form of a data stream; in another case, the upstream computing node may, after completing the execution of the above-mentioned first subtask, simply divide the output data generated by executing the first subtask into multiple data slices corresponding one-to-one to the multiple downstream computing nodes executing the above-mentioned second subtask, but wait until receiving the data acquisition request sent by the downstream computing node, and then use a unified data structure to further abstract each data slice divided from the output data generated by executing the first subtask into the form of a data stream.

[0111] In some embodiments, based on the above unified data structure, after each data shard is abstracted into the form of a data stream, each data shard can specifically contain at least one batch of data, and any batch of data can specifically contain at least one data block. For example, see Figure 6 , Figure 6 This is a data structure diagram of data generated by an upstream computing node executing a first subtask as shown in this specification.

[0112] like Figure 6 As shown, the data structure of the data generated by the upstream computing node executing the first subtask may specifically include Slice, batch, and Message.

[0113] The above Slice is the basic management unit of the data generated by executing the first subtask. Usually, the upstream computing node will divide the data generated by executing the upstream task into N slices according to the concurrency N of the downstream computing node executing the above second subtask. Each slice can also be divided into Figure 5 The unified data structure shown is further abstracted into the form of data stream.

[0114] A batch, as mentioned above, refers to the basic management unit of data within a slice. A batch represents a batch of data, and a slice can contain one or more batches of data. In other words, a slice can be a data stream consisting of one or more batches of data. Within a slice, different batches can be separated by barriers. Barriers can be thought of as delimiters that distinguish one batch from another and indicate the end of the current batch. A barrier can also record metadata about the current batch; for example, this metadata may include information such as the batch ID and batch length.

[0115] It should be noted that the number of data batches contained within a slice that an upstream compute node allocates for any downstream compute node is typically a number corresponding to the computing mode used by the downstream compute node to execute the downstream task. In other words, the number of data batches contained within a slice that an upstream compute node allocates for a downstream compute node typically depends on the computing mode used by the downstream compute node to execute the downstream task.

[0116] For example, in a distributed graph computing scenario, the above-mentioned multiple computing modes may specifically include stream processing mode, batch processing mode, and iterative graph computing mode.

[0117] Among them, in stream processing mode, a Slice can contain multiple batches of data. In batch processing mode, a Slice can only contain a single batch of data. For iterative graph processing mode, in actual applications, it can be executed in a stream processing manner or in a batch processing manner. On the one hand, if the iterative graph computing mode is an iterative graph computing mode executed in a stream processing manner, a Slice can contain multiple batches of data; if the iterative graph computing mode is an iterative graph computing mode executed in a batch processing manner, a Slice can only contain a single batch of data.

[0118] In actual applications, after the upstream computing node divides a slice for any downstream computing node, it can further obtain the computing mode adopted by the downstream computing node when executing the second subtask, and then determine the number of batches of data that should be contained in the slice divided for the downstream computing node based on the obtained computing mode, and further divide the data contained in the slice into one or more batches of data based on the determined number, and then further abstract the data contained in the slice divided for the downstream computing node into a data flow.

[0119] For example, in one case, if the upstream computing node uses the above-mentioned unified data structure after completing the execution of the above-mentioned first subtask, each data slice divided from the output data generated by executing the first subtask will be further abstracted into the form of a data stream. At this time, the computing mode adopted by the downstream computing node to execute the second subtask can be obtained from the task scheduler (this computing mode is usually a computing mode planned in advance when specifying the execution plan of the graph computing task). Then, based on the obtained computing mode, the number of batches of data that should be contained in the slice divided for the downstream computing node can be determined, and the data contained in the slice can be further divided into one or more batches of data based on the determined number.

[0120] In another case, if the upstream computing node, after completing the execution of the above-mentioned first subtask, only divides the output data generated by the execution of the first subtask into multiple data slices corresponding to the multiple downstream computing nodes executing the above-mentioned second subtask, and after receiving the data acquisition request sent by the downstream computing node, uses a unified data structure to further abstract each data slice divided from the output data generated by the execution of the first subtask into the form of a data stream, then in the data acquisition stage, the computing mode adopted by the downstream computing node to execute the second subtask can be determined according to the data acquisition conditions contained in the data acquisition request sent by the downstream computing node, and then based on the acquired computing mode, the number of batches of data that should be contained in the slice divided for the downstream computing node can be determined, and the data contained in the slice can be further divided into one or more batches of data based on the determined number.

[0121] The Message refers to a basic management unit of data contained in a batch. One Message can represent a smaller data block obtained by dividing the data contained in a batch. The data contained in a single batch can be further divided into one or more Messages. That is, a batch can also be a data stream composed of one or more Messages. In addition to recording part of the data contained in a batch, some metadata of the current batch can also be recorded in a Message in actual application. For example, batch ID, length of the batch, and the like. It should be noted that the number of Messages obtained by dividing the data contained in a batch depends on the total length of the data contained in the batch and the unit length set for each Message. In actual application, the unit length of a Message can be set flexibly based on specific requirements, and the data contained in a batch can be divided based on the unit length.

[0122] In some embodiments, the data distribution manner provided by the above-mentioned multiple computing modes in the distributed system can be unified into a Pull-based Shuffle data distribution manner. That is, the downstream computing node actively sends a data acquisition request to the upstream computing node, and the upstream computing node can process the data acquisition request and deliver the data required by the downstream computing node to the downstream computing node on demand.

[0123] In this case, for any target downstream computing node in the multiple downstream computing nodes for executing the second subtask, the data acquisition condition corresponding to the computing mode adopted for executing the second subtask can be determined first.

[0124] The data acquisition condition is specifically used to acquire the input data required by the target downstream computing node for executing the second subtask from the data generated by the upstream computing nodes for executing the first subtask. Different computing modes adopted by the target downstream computing node for executing the second subtask can correspond to different data acquisition conditions. That is, the data acquisition condition can be different in different computing modes. The upstream computing node should be able to accurately determine the computing mode adopted by the downstream computing node for executing the second subtask according to the difference in the data acquisition condition contained in the data acquisition request sent by the downstream computing node.

[0125] When the target downstream computing node determines the data acquisition condition corresponding to the computing mode adopted by itself to execute the second subtask, the data acquisition request can be constructed for the plurality of upstream computing nodes for executing the first subtask based on the determined data acquisition condition; that is, based on the concurrency of the upstream computing nodes for executing the first subtask, the plurality of data acquisition requests are constructed; for example, assuming that the concurrency of the upstream computing nodes for executing the first subtask is N, then N data acquisition requests need to be constructed.

[0126] Then, the constructed data acquisition requests can be respectively sent to the plurality of upstream computing nodes.

[0127] In some embodiments, in different computing modes, the upstream computing nodes for executing the first subtask and the downstream computing nodes for executing the second subtask can run simultaneously or sequentially.

[0128] On the one hand, if the upstream computing nodes for executing the first subtask and the downstream computing nodes for executing the second subtask run simultaneously, the downstream computing nodes for executing the second subtask can send the data acquisition request to the upstream computing nodes immediately after the upstream computing nodes and the downstream computing nodes are started simultaneously.

[0129] On the other hand, if the upstream computing nodes for executing the first subtask run prior to the downstream computing nodes for executing the second subtask, the downstream computing nodes for executing the second subtask can further send the data acquisition request to the upstream computing nodes after the upstream computing nodes for executing the second subtask are completed.

[0130] For example, in one example, in the batch processing mode, since the downstream computing nodes are usually scheduled to continue to execute the second subtask after the upstream computing nodes complete the execution of the first subtask, based on the characteristics of the task execution in the batch processing mode, the upstream computing nodes are usually started prior to the downstream computing nodes, and when the upstream computing nodes complete the execution of the first subtask, the downstream computing nodes are started; in this case, the downstream computing nodes can send the data acquisition request to the upstream computing nodes after the upstream computing nodes complete the execution of the first subtask and the downstream computing nodes are started.

[0131] In another example, in the stream processing mode, the upstream computing nodes and the downstream computing nodes are usually started simultaneously, in which case the downstream computing nodes can send the data acquisition request to the upstream computing nodes immediately after the upstream computing nodes and the downstream computing nodes are started simultaneously.

[0132] In the third example, in iterative graph computing mode, since the upstream and downstream compute nodes typically correspond to the same compute node, they can be two different threads running on the same compute node, and these two threads can typically be started simultaneously. In this case, the thread corresponding to the downstream compute node can, after simultaneously starting with the threads corresponding to the upstream and downstream compute nodes, immediately send a data acquisition request to the thread corresponding to the upstream compute node.

[0133] Step 204: In response to the data acquisition request, acquire target data that meets the data acquisition condition from the data generated by executing the first subtask;

[0134] For any target upstream computing node among the multiple upstream computing nodes that execute the above-mentioned first subtask, it can receive a data acquisition request sent by the downstream computing node. After receiving the data acquisition request sent by any target downstream computing node among the above-mentioned multiple downstream computing nodes, it can further respond to the data acquisition request and adopt a unified Pull-based Shuffle data distribution method for different computing modes to distribute data to the target downstream computing node.

[0135] Among them, it should be noted that, since a unified Pull-basedShuffle data distribution method can be adopted for different computing modes to distribute data to downstream computing nodes, no matter what computing mode the downstream computing node adopts, the upstream computing node can adopt a unified data distribution method to distribute data to the downstream computing; therefore, in this case, multiple downstream computing nodes used to execute the second subtask can adopt different computing modes to execute the second subtask in parallel.

[0136] In some embodiments, the unified Pull-based Shuffle data distribution method provided for the above-mentioned multiple computing modes may specifically be a non-blocking pull mode.

[0137] In this non-blocking pull mode, the data acquisition request sent by the downstream computing node to the upstream computing node can be an asynchronous request. Accordingly, upon receiving the data acquisition request from the target downstream computing node, the target upstream computing node can asynchronously respond to the data acquisition request and acquire the target data that meets the data acquisition condition from the data generated by executing the first subtask.

[0138] Among them, it should be noted that the specific form of the data acquisition conditions contained in the above-mentioned data acquisition request will no longer be specifically limited in this specification. In actual applications, the above-mentioned data acquisition request can be used as the above-mentioned data acquisition condition by explicitly carrying the shard identifier of the data shard assigned to the downstream computing node and the batch identifier of the data batch to be obtained or the batch quantity of the batch to be obtained. It can also be used as the above-mentioned data acquisition condition by carrying some implicit data acquisition rules in the above-mentioned data acquisition request, which will no longer be listed one by one in this specification.

[0139] In some embodiments, the data acquisition conditions included in the above-mentioned data acquisition request may specifically include the shard identifier of the data shard allocated by the above-mentioned target upstream computing node to the above-mentioned target downstream computing node, the batch identifier of the starting data batch to be acquired, and the batch quantity of the data batch to be acquired.

[0140] In this case, after receiving the data acquisition request sent by the target downstream computing node, the target upstream computing node can first obtain the shard identifier, batch identifier, and batch quantity contained in the data acquisition request as data acquisition conditions.

[0141] Secondly, the target data shard corresponding to the shard identifier may be determined from the data shards respectively allocated to the plurality of downstream computing nodes;

[0142] For example, in actual applications, when an upstream computing node allocates a data shard to a downstream computing node, it may use the node identifier of the downstream computing node as the shard identifier of the data shard by default, or it may use the node identifier of the downstream computing node as part of the shard identifier of the data shard by default. In this way, when the target downstream computing node constructs a data acquisition request, it may use its own node identifier as the shard identifier of the data shard assigned to it by the target upstream computing node and carry it in the data acquisition request by default. After receiving the data acquisition request, the target upstream computing node may directly match the shard identifier contained in the data acquisition request with the shard identifiers of the data shards that have been allocated to each downstream computing node, and thus determine the target data shard corresponding to the shard identifier contained in the data acquisition request.

[0143] Then, a target data batch corresponding to the batch identifier may be further determined from at least one batch of data contained in the target data shard.

[0144] For example, in actual applications, the batch identifier of each batch of data included in the data shards allocated by the upstream computing node to the downstream computing node may be a monotonically increasing identifier starting from a default value (e.g., 0 or 1) as a starting value. In this case, the batch identifier of the starting data batch included in the data acquisition request sent by the target downstream computing node to the target upstream computing node may be the default value serving as the starting value. In this case, the default value represents the batch identifier corresponding to the first batch of data included in the target data shard. Upon receiving the data acquisition request, the target upstream computing node may, by default, determine the first batch of data included in the target data shard as the target data batch corresponding to the batch identifier.

[0145] Finally, the target data batch can be used as the starting data batch, and at least one batch of data corresponding to the value of the above batch number can be obtained in sequence as the target data; for example, if the value of the above batch number is N, it is necessary to start from the target data batch and obtain N batches of data in sequence as the target data; then, the obtained target data can be sent to the above target downstream computing node.

[0146] For example, still taking the above-mentioned multiple computing modes including stream processing mode, batch processing mode, and iterative graph computing mode as an example, if the computing mode adopted by the above-mentioned target downstream computing node to execute the above-mentioned second subtask is stream processing mode, then the target data shard allocated by the above-mentioned target upstream computing node to the target downstream computing node may specifically include multiple batches of data; in this case, the batch identifier of the above-mentioned starting data batch included in the data acquisition request sent by the target downstream computing node to the above-mentioned target upstream computing node may specifically be the batch identifier corresponding to the first batch of data among the multiple batches of data included in the target data shard; for example, the batch identifiers of the multiple batches of data included in the target data shard may specifically be identifiers that monotonically increase starting from a default value as the starting value, in which case the identifier of the first batch of data may be the default value as the starting value. The value of the above-mentioned batch number may be infinite (for example, it may be a character representing infinity), indicating that all batches of data included in the target data shard are to be acquired.

[0147] If the computing mode adopted by the above-mentioned target downstream computing node to execute the above-mentioned second subtask is batch processing mode, then the target data shard allocated by the above-mentioned target upstream computing node to the target downstream computing node may specifically contain only a single batch of data; in this case, the batch identifier of the above-mentioned starting data batch contained in the data acquisition request sent by the target downstream computing node to the above-mentioned target upstream computing node may specifically be a batch identifier corresponding to the only batch of data; for example, the batch identifier corresponding to the only batch of data may be a default value (such as 0 or 1) serving as a starting value; the value of the above-mentioned batch quantity may be 1, indicating that in batch processing mode, it is only necessary to obtain the only batch of data contained in the above-mentioned target data shard.

[0148] If the target downstream computing node executes the second subtask in an iterative graph processing mode, then the second subtask itself is an iterative graph computing subtask. In this case, since iterative graph processing modes generally include iterative graph computing modes executed in a stream processing manner and iterative graph computing modes executed in a batch processing manner, these two situations need to be discussed separately:

[0149] On the one hand, if the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a stream processing manner, then the first subtask can generally be a computing task of any iterative round included in the iterative graph computing, and the second subtask can generally be a computing task of the next iterative round of the iterative round. The target data shard allocated by the target upstream computing node to the target downstream computing node can specifically include multiple batches of data. In other words, in the iterative graph computing mode executed in a stream processing manner, the data contained in the target data shard will be further divided into multiple batches of data and sent to the target downstream computing node in multiple batches.

[0150] In this case, the batch identifier of the initial data batch included in the data acquisition request sent by the target downstream computing node to the target upstream computing node may specifically be the batch identifier corresponding to the first batch of data among the multiple batches of data included in the target data shard. For example, the batch identifiers of the multiple batches of data included in the target data shard may specifically be identifiers that monotonically increase from a default value serving as the starting value. In this case, the identifier of the first batch of data may be the default value serving as the starting value. The value of the batch number may be infinite.

[0151] On the other hand, if the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a batch manner, then the first subtask can still be a computing task of any iterative round included in the iterative graph computing, and the second subtask can usually be a computing task of the next iterative round of the iterative round. The target data shard allocated by the target upstream computing node to the target downstream computing node can specifically contain only a single batch of data. In other words, in the iterative graph computing mode executed in a batch manner, the data contained in the target data shard will be further divided into a single batch of data, and then sent to the target downstream computing node in batches.

[0152] In this case, the batch identifier of the above-mentioned starting data batch contained in the data acquisition request sent by the target downstream computing node to the above-mentioned target upstream computing node can specifically be a batch identifier corresponding to the only batch of data; for example, the batch identifier corresponding to the only batch of data can be a default value (such as 0 or 1) serving as a starting value; the value of the above-mentioned batch quantity can be 1, indicating that in the iterative graph computing mode executed in a batch processing manner, only the only batch of data contained in the above-mentioned target data shard needs to be obtained.

[0153] In some embodiments, the data acquisition condition included in the above data acquisition request may further include the shard identifier of the data shard allocated by the above target upstream computing node to the above target downstream computing node, and the maximum batch identifier of all data batches that need to be acquired.

[0154] In this case, after receiving the data acquisition request sent by the target downstream computing node, the target upstream computing node may first obtain the shard identifier and the maximum batch identifier contained in the data acquisition request as data acquisition conditions.

[0155] Secondly, it is also possible to determine the target data shard corresponding to the above-mentioned shard identifier from the data shards respectively allocated to the above-mentioned multiple downstream computing nodes, and then obtain at least one batch of data with a batch identifier not greater than the above-mentioned maximum batch identifier from at least one batch of data contained in the above-mentioned target data shard as the above-mentioned target data.

[0156] For example, still taking the above-mentioned multiple computing modes including stream processing mode, batch processing mode and iterative graph computing mode as an example, if the computing mode adopted by the above-mentioned target downstream computing node to execute the above-mentioned second subtask is stream processing mode, then the above-mentioned target data shard may contain multiple batches of data; for example, the batch identifiers of the multiple batches of data contained in the target data shard may specifically be an identifier that monotonically increases from a default value (such as 0 or 1) as a starting value; the value of the above-mentioned maximum batch identifier may be infinity, indicating that all batches of data contained in the target data shard are to be obtained.

[0157] If the computing mode adopted by the above-mentioned target downstream computing node to execute the second subtask is batch processing mode, then the above-mentioned target data shard may only contain a single batch of data; for example, the batch identifier corresponding to the single batch of data may be a default value (such as 0 or 1) as a starting value; the value of the above-mentioned maximum batch identifier may be the batch identifier corresponding to the single batch of data.

[0158] If the target downstream computing node executes the second subtask in an iterative graph processing mode, then the second subtask itself is an iterative graph computing subtask. In this case, since iterative graph processing modes generally include iterative graph computing modes executed in a stream processing manner and iterative graph computing modes executed in a batch processing manner, these two situations need to be discussed separately:

[0159] On the one hand, if the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a stream processing manner, then the first subtask is a computing task of any one round of iteration in the iterative graph computing, and the second subtask is a computing task of the next round of iteration of the any one round of iteration; the target data shard may contain multiple batches of data, and the value of the maximum batch identifier may be infinite, indicating that the number of all batches contained in the target data shard is to be obtained;

[0160] On the other hand, if the computing mode adopted by the above-mentioned target downstream computing node to execute the above-mentioned second subtask is an iterative graph computing mode executed in a batch processing manner, then the above-mentioned first subtask is still a computing task of any round of iteration in the iterative graph calculation, and the above-mentioned second subtask is a computing task of the next round of iteration of any round of iteration; the above-mentioned target data shard may contain only one batch of data, and the value of the above-mentioned maximum batch identifier may be a batch identifier corresponding to the only batch of data.

[0161] Step 206: Send the target data to the target downstream computing node, so that the target downstream computing node continues to execute the second subtask using the target data as input data.

[0162] After obtaining target data that meets the data acquisition conditions from the data generated by executing the first subtask, the target upstream node may send the target data to the target downstream computing node. The target data may typically include at least one batch of data obtained from the data shards allocated to the target downstream computing node.

[0163] After receiving the target data sent by the target upstream node, the target downstream computing node may use the target data as input data to continue executing the second subtask.

[0164] It should be noted that, since the multiple upstream computing nodes that execute the first subtask in parallel will respectively allocate a data slice to the target downstream computing node from the data generated by executing the first subtask, the above-mentioned target downstream computing node can specifically obtain the target data from the data slices allocated to itself by each upstream computing node in the multiple upstream computing nodes by sending a data acquisition request to the multiple upstream computing nodes, and then integrate the multiple target data obtained from the data slices allocated to itself by each upstream computing node as input data to continue to execute the above-mentioned second subtask.

[0165] For example, see Figure 5 For downstream computing node A, slice1-A generated by upstream computing node 1 for downstream computing node A, slice2-A generated by upstream computing node 2 for downstream computing node A, and slice3-A generated by upstream computing node 3 for downstream computing node A can be obtained respectively, and then slice1-A, slice2-A and slice3-A are integrated as input data to continue to execute the above second subtask.

[0166] In some embodiments, since a unified Pull-based Shuffle data distribution method can be used for different computing modes to distribute data to downstream computing nodes, and a unified data structure can also be used to further abstract the data slices allocated to each downstream subtask from the output data generated by executing the upstream task into the form of a data stream; therefore, adopting this unified data distribution mode is equivalent to abstracting the batch data that needs to be processed in the batch processing mode into a data stream of batch 1, and abstracting the data generated by N iterations that need to be processed in the iterative graph computing mode into a finite number of batches of data streams.

[0167] Based on this feature, when the above-mentioned target upstream computing node sends the acquired target data to the target downstream computing node, it can specifically send the data blocks of at least one batch of data contained in the acquired target data to the above-mentioned target downstream computing node in a streaming manner.

[0168] For example, see Figure 6 , each message in at least one slice contained in the acquired target data can be sent one by one to the target downstream computing node in a streaming manner, with the message as the smallest transmission unit.

[0169] In some embodiments, since in the iterative graph computing mode, the above-mentioned target upstream computing node and the above-mentioned target downstream computing node usually correspond to the same computing node, at this time the above-mentioned target upstream computing node and the above-mentioned target downstream computing node can usually be two different threads running on the same computing node. Therefore, if the computing mode adopted by the above-mentioned target downstream computing node to execute the second subtask is the iterative graph computing mode, at this time the above-mentioned first subtask is usually the computing task of any round of iteration in the iterative graph computing, and the above-mentioned second subtask is the computing task of the next round of iteration of this any round of iteration.

[0170] In this case, when the thread corresponding to the above-mentioned target upstream computing node sends the obtained target data to the target downstream computing node, it can specifically send the above-mentioned target data to a pre-designated local storage space on the computing node; for example, the data blocks of at least one batch of data contained in the obtained target data can be written one by one into the above-mentioned local storage space in a streaming manner.

[0171] The thread corresponding to the target downstream computing node may specifically read the target data from the local storage space as input data for executing the second subtask.

[0172] The local storage space may be a memory space or a disk storage space pre-designated on the computing node.

[0173] In some embodiments, if the computing mode adopted by the above-mentioned target downstream computing node to execute the downstream task is a batch processing mode, at this time if the concurrency of the upstream task and the downstream task is the same, the second subtask executed by the above-mentioned target downstream computing node and the first subtask executed by the above-mentioned target upstream computing node can be deployed to the same computing node for execution; that is, in the batch processing mode where the concurrency of the upstream task and the downstream task is the same, the above-mentioned target upstream computing node and the above-mentioned target downstream computing node can correspond to the same computing node, and at this time the above-mentioned target upstream computing node and the above-mentioned target downstream computing node can be two different threads running on the same computing node.

[0174] In this case, since the upstream computing node is usually started before the downstream computing node in batch mode, after the thread corresponding to the above-mentioned target upstream computing node is started, an additional independent thread can be started in advance on the computing node for the thread corresponding to the above-mentioned target downstream computing node; wherein, the independent thread will run independently of the thread corresponding to the above-mentioned target downstream computing node. After the independent thread is started, data acquisition requests can be sent in advance to multiple upstream computing nodes used to execute the above-mentioned first subtask.

[0175] In this way, since the thread corresponding to the target upstream computing node can receive the data acquisition request sent by the thread corresponding to the target downstream computing node in advance, the thread corresponding to the target upstream computing node can obtain the target data from the data slice allocated to the thread corresponding to the target upstream computing node in advance, and can also send the obtained target data to a pre-designated local storage space on the computing node in advance. Subsequently, when the thread corresponding to the target downstream computing node is started, it can directly read the target data from the local storage space as input data for executing the second subtask, without having to repeatedly send data acquisition requests.

[0176] See Figure 7 , Figure 7 This is a flowchart of another data distribution method for distributed graph computing shown in this specification; computing nodes in a distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; wherein the first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system; the method is applied to any target downstream computing node among multiple downstream computing nodes executing the second subtask; the method includes the following execution process:

[0177] Step 702: Determine a data acquisition condition based on the computing mode used to execute the second subtask; wherein the data acquisition condition is used to obtain input data required for executing the second subtask from any target upstream computing node among the multiple upstream computing nodes executing the first subtask;

[0178] Step 704: Generate a data acquisition request including the data acquisition condition, and send the data acquisition request to the target upstream computing node, so that the target upstream computing node acquires target data that meets the data acquisition condition from the data generated by executing the first subtask.

[0179] Step 704: Receive the target data sent by the target upstream computing node, and use the target data as input data to continue executing the second subtask.

[0180] It should be noted that the above Figure 2 The execution flow shown in FIG20 includes steps 202 to 204, which is a method flow in which any target upstream computing node among the multiple downstream computing nodes that execute the first subtask (that is, the upstream task) is used as the execution subject. Figure 2 The difference is, Figure 7 The execution flow shown in FIG. 702 includes steps 702 to 704, which is a method flow in which any target downstream computing node among the multiple downstream computing nodes that execute the second subtask is used as the execution subject. Figure 7 The implementation details of the method flow shown in Figure 2 The implementation details of the method flow shown in FIG are exactly the same, so in this embodiment, no further Figure 7 The implementation details of each step included in the embodiment are described in detail, and those skilled in the art can refer to the description of the previous embodiments.

[0181] In the above technical solution, a unified data distribution method can be provided for the various computing modes supported by the distributed system that performs distributed graph computing. For different computing modes, there is no need to provide different data distribution methods separately, which can reduce the complexity of system development and maintenance and allow the system to adapt to different computing modes more flexibly.

[0182] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0183] Figure 8 This is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 8At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include other required hardware. One or more embodiments of this specification can be implemented based on software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into the memory 808 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0184] like Figure 9 As shown, Figure 9 This is a block diagram of a data distribution device for distributed graph computing according to an exemplary embodiment of the present specification. The device can be operated on Figure 8 The electronic device shown in the figure implements the technical solution of this specification. The computing nodes in the distributed system for executing a distributed graph computing task support executing the distributed graph computing task in multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; the first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system; the device is applied to any target upstream computing node among the multiple upstream computing nodes executing the first subtask; the device 90 includes:

[0185] A receiving module 901 receives a data acquisition request sent by any target downstream computing node among multiple downstream computing nodes for executing the second subtask; wherein the data acquisition request includes a data acquisition condition determined by the target downstream computing node based on the computing mode adopted for executing the second subtask; the data acquisition condition is used to obtain input data required for the target downstream computing node to execute the second subtask from data generated by the target upstream computing node when executing the first subtask; different computing modes adopted by the target downstream computing node to execute the second subtask correspond to different data acquisition conditions;

[0186] An acquisition module 902, in response to the data acquisition request, acquires target data that meets the data acquisition condition from the data generated by executing the first subtask;

[0187] The sending module 903 sends the target data to the target downstream computing node, so that the target downstream computing node continues to execute the second subtask using the target data as input data.

[0188] like Figure 10 As shown, Figure 10 This is a block diagram of another data distribution device for distributed graph computing according to an exemplary embodiment of the present specification. The device can also be operated on Figure 8 The electronic device shown in the figure implements the technical solution of this specification. The computing nodes in the distributed system for executing a distributed graph computing task support executing the distributed graph computing task in multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; the first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system; the method is applied to any target downstream computing node among multiple downstream computing nodes executing the second subtask; the apparatus 100 includes:

[0189] Determining module 1001, determining a data acquisition condition based on a computing mode used to execute the second subtask; wherein the data acquisition condition is used to obtain input data required for executing the second subtask from any target upstream computing node among multiple upstream computing nodes executing the first subtask;

[0190] A sending module 1002 generates a data acquisition request including the data acquisition condition, and sends the data acquisition request to the target upstream computing node, so that the target upstream computing node acquires target data that meets the data acquisition condition from data generated by executing the first subtask;

[0191] The execution module 1003 receives the target data sent by the target upstream computing node, and uses the target data as input data to continue executing the second subtask.

[0192] Accordingly, this specification also provides an electronic device, which includes a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps in all the method flows described above.

[0193] Accordingly, this specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0194] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0195] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using physical hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly performed using software called a "logic compiler." This is similar to the software compilers used during program development. Before compilation, the original code must be written in a specific programming language, called a Hardware Description Language (HDL). There are many types of HDL, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that simply by programming a method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0196] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller purely in computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the means for implementing the various functions included therein can also be considered as structures within the hardware component. Alternatively, the means for implementing the various functions can be considered both a software module implementing the method and a structure within the hardware component.

[0197] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the future development of computer technology, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0198] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is only one way of executing the steps among many, and does not represent the only execution order. When a device or terminal product is actually executed, the method sequence shown in the embodiments or figures can be executed sequentially or in parallel (for example, in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, product, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, product, or device. Without further limitation, it does not exclude the presence of other identical or equivalent elements in the process, method, product, or device that includes the elements. For example, if words such as first and second are used to indicate names, they do not indicate any particular order.

[0199] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0200] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0203] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0204] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0205] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0206] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as methods, systems, or computer program products. Thus, one or more embodiments of this specification may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0207] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0208] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0209] The foregoing is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification shall be included within the scope of the claims.

Claims

1. A data distribution method for distributed graph computing, wherein computing nodes in a distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing tasks include a first subtask as an upstream task and a second subtask as a downstream task; wherein, The first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system; The method is applied to any target upstream computing node among multiple upstream computing nodes that execute the first subtask, and the method includes: receiving a data acquisition request sent by any target downstream computing node among a plurality of downstream computing nodes for executing the second subtask; wherein the data acquisition request includes a data acquisition condition determined by the target downstream computing node based on a computing mode adopted for executing the second subtask; the data acquisition condition is used to acquire input data required for the target downstream computing node to execute the second subtask from data generated by the target upstream computing node when executing the first subtask; different computing modes adopted by the target downstream computing node to execute the second subtask correspond to different data acquisition conditions; In response to the data acquisition request, acquiring target data that meets the data acquisition condition from the data generated by executing the first subtask; The target data is sent to the target downstream computing node, so that the target downstream computing node continues to execute the second subtask using the target data as input data.

2. The method of claim 1, wherein the data generated by executing the first subtask comprises data shards respectively allocated to multiple downstream computing nodes executing the second subtask; the data shards comprise at least one batch of data; and any batch of data comprises at least one data block; in, The number of batches of data contained in the data shards allocated to any downstream computing node is the number corresponding to the computing mode adopted by the downstream computing node to execute the second subtask.

3. The method of claim 2, wherein the data acquisition condition comprises a shard identifier of a data shard allocated by the target upstream computing node to the target downstream computing node, a batch identifier of a starting data batch to be acquired, and a batch quantity of the data batch to be acquired; In response to the data acquisition request, acquiring target data that meets the data acquisition condition from the data generated by executing the first subtask includes: In response to the data acquisition request, determining a target data shard corresponding to the shard identifier from the data shards respectively allocated to the plurality of downstream computing nodes; From at least one batch of data contained in the target data shard, determine a target data batch corresponding to the batch identifier, use the target data batch as the starting data batch, and sequentially obtain at least one batch of data corresponding to the value of the batch quantity as the target data.

4. The method of claim 3, wherein the multiple computing modes include a stream processing mode, a batch processing mode, and an iterative graph computing mode; If the computing mode adopted by the target downstream computing node to execute the second subtask is stream processing mode, the target data shard contains multiple batches of data, the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data, and the value of the batch number is infinite; If the computing mode adopted by the target downstream computing node to execute the second subtask is batch mode, the target data shard contains only one batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the only batch of data, and the value of the batch quantity is 1; in, The batch identifiers of the multiple batches of data are monotonically increased.

5. The method of claim 4, wherein the iterative graph computing mode comprises an iterative graph computing mode executed in a stream processing manner; and an iterative graph computing mode executed in a batch processing manner; If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a stream processing manner, the first subtask is a computing task of any one round of iteration in the iterative graph computing, and the second subtask is a computing task of the next round of iteration of the any one round of iteration; the target data shard contains multiple batches of data; the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data; and the value of the batch quantity is infinite; If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a batch processing manner, the first subtask is a computing task of any iterative round in the iterative graph computing, and the second subtask is a computing task of the next iterative round of the iterative round; the target data shard contains only one batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the only batch of data, and the value of the batch quantity is 1; The batch identifiers of the multiple batches of data are monotonically increasing.

6. The method according to claim 2, wherein the data acquisition condition includes a shard identifier of a data shard allocated by the target upstream computing node to the target downstream computing node, and a maximum batch identifier of all data batches to be acquired; In response to the data acquisition request, acquiring target data that meets the data acquisition condition from the data generated by executing the first subtask includes: In response to the data acquisition request, determining a target data shard corresponding to the shard identifier from the data shards respectively allocated to the plurality of downstream computing nodes; At least one batch of data whose batch identifier is not greater than the maximum batch identifier is obtained from at least one batch of data included in the target data shard as the target data.

7. The method of claim 6, wherein the multiple computing modes include a stream processing mode, a batch processing mode, and an iterative graph computing mode; If the computing mode adopted by the target downstream computing node to execute the second subtask is stream processing mode, the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinite; If the computing mode adopted by the target downstream computing node to execute the second subtask is batch processing mode, and the target data shard contains data of only one batch, the value of the maximum batch identifier is the batch identifier corresponding to the only batch of data; in, The batch identifiers of the multiple batches of data are monotonically increased.

8. The method of claim 7, wherein the iterative graph computing mode comprises an iterative graph computing mode executed in a stream processing manner; and an iterative graph computing mode executed in a batch processing manner; If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a stream processing manner, the first subtask is a computing task of any iterative round in the iterative graph computing, and the second subtask is a computing task of the next iterative round of any iterative round; the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinite; If the computing mode adopted by the target downstream computing node to execute the second subtask is an iterative graph computing mode executed in a batch processing manner, the first subtask is a computing task of any iterative round in the iterative graph computing, and the second subtask is a computing task of the next iterative round of the iterative round; the target data shard contains only one batch of data, and the value of the maximum batch identifier is the batch identifier corresponding to the only batch of data; The batch identifiers of the multiple batches of data are monotonically increasing.

9. The method according to claim 2, wherein the data acquisition request is an asynchronous request; In response to the data acquisition request, acquiring target data that meets the data acquisition condition from the data generated by executing the first subtask includes: Asynchronously responding to the data acquisition request, acquiring target data that meets the data acquisition condition from the data generated by executing the first subtask.

10. The method of claim 9, wherein the target data comprises at least one batch of data obtained from a data shard allocated to the target downstream computing node; Sending the target data to the target downstream computing node includes: The data blocks contained in the acquired at least one batch of data are sent to the target downstream computing node in a streaming manner. 11 . The method of claim 1 , wherein the plurality of downstream computing nodes for executing the second subtask adopt different computing modes to execute the second subtask.

12. A data distribution method for distributed graph computing, wherein computing nodes in a distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing tasks include a first subtask as an upstream task and a second subtask as a downstream task; wherein, The first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system; The method is applied to any target downstream computing node among multiple downstream computing nodes that execute the second subtask, and the method includes: Determining a data acquisition condition based on a computing mode used to execute the second subtask; wherein the data acquisition condition is used to obtain input data required for executing the second subtask from any target upstream computing node among multiple upstream computing nodes executing the first subtask; generating a data acquisition request including the data acquisition condition, and sending the data acquisition request to the target upstream computing node, so that the target upstream computing node acquires target data that meets the data acquisition condition from data generated by executing the first subtask; Receive the target data sent by the target upstream computing node, and use the target data as input data to continue executing the second subtask.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images