Data distribution method and device for distributed graph calculation

Abstract

The invention discloses a data distribution method for distributed graph calculation. Computing nodes in a distributed system support execution of distributed graph calculation tasks according to multiple calculation modes. The distributed graph calculation task comprises a first subtask serving as an upstream task and a second subtask serving as a downstream task; the first subtask and the second subtask are both concurrently executed by a plurality of computing nodes; the method comprises the following steps: receiving a data acquisition request sent by any target downstream computing node in a plurality of downstream computing nodes for executing a second subtask; the data acquisition request comprises a data acquisition condition determined by the target downstream computing node on the basis of a computing mode adopted for executing the second subtask; different calculation modes adopted by the target downstream calculation node for executing the second subtask correspond to different data acquisition conditions respectively; acquiring target data meeting a data acquisition condition from data generated by executing the first subtask; and issuing the target data to the target downstream computing node.

Description

[0001] This application is a divisional application of Chinese invention patent application filed on June 12, 2025, with application number 2025107874565 and invention title "A data distribution method and apparatus for distributed graph computing". Technical Field

[0002] The embodiments in this specification pertain to the field of graph computing, and particularly relate to a data distribution method and apparatus for distributed graph computing. Background Technology

[0003] To adapt to diverse business needs, more and more graph computing engines are beginning to support the execution of graph computing tasks in various different computing modes; for example, graph computing engines can support the execution of graph computing tasks in stream processing mode, batch processing mode, and graph processing mode.

[0004] In distributed graph computing scenarios, regardless of the mode in which the graph computing engine executes graph computation, data shuffling between upstream and downstream tasks is unavoidable. Data shuffling refers to the process by which upstream tasks in distributed graph computing distribute data to downstream tasks via a network.

[0005] However, in practical applications, the shuffle methods used by upstream and downstream tasks during data shuffle often differ under different computing modes. This means that graph computing engines need to provide different data shuffle mechanisms for different computing modes, which may lead to increased system complexity and excessive system overhead. Summary of the Invention

[0006] This specification proposes a data distribution method for distributed graph computing, wherein computing nodes in a distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; wherein both the first subtask and the second subtask are executed concurrently by multiple computing nodes in the distributed system; the method is applied to any target upstream computing node among the multiple upstream computing nodes executing the first subtask, and the method includes:

[0007] The system receives a data acquisition request from any target downstream computing node among a plurality of downstream computing nodes used to execute the second subtask; wherein the data acquisition request includes data acquisition conditions determined by the target downstream computing node based on the computing mode adopted to execute the second subtask; the data acquisition conditions are used to obtain the input data required by the target downstream computing node to execute the second subtask from the data generated by the target upstream computing node executing the first subtask; different computing modes adopted by the target downstream computing node to execute the second subtask correspond to different data acquisition conditions;

[0008] In response to the data acquisition request, target data that meets the data acquisition conditions is obtained from the data generated by executing the first subtask;

[0009] The target data is sent to the target downstream computing node, so that the target downstream computing node can use the target data as input data to continue executing the second subtask.

[0010] Optionally, the data generated from executing the first subtask includes data shards allocated to the multiple downstream computing nodes executing the second subtask; the data shards contain at least one batch of data; any batch of data contains at least one data block;

[0011] The number of data batches contained in the data shard allocated to any downstream computing node is the number corresponding to the computing mode adopted by the downstream computing node in executing the second subtask.

[0012] Optionally, the data acquisition conditions include the fragment identifier of the data fragment allocated by the target upstream computing node to the target downstream computing node, the batch identifier of the starting data batch to be acquired, and the batch number of the data batch to be acquired.

[0013] In response to the data acquisition request, target data that meets the data acquisition conditions is acquired from the data generated during the execution of the first subtask, including:

[0014] In response to the data acquisition request, a target data shard corresponding to the shard identifier is determined from the data shards allocated to the plurality of downstream computing nodes respectively;

[0015] From at least one batch of data contained in the target data segment, determine the target data batch corresponding to the batch identifier, and use the target data batch as the starting data batch. Sequentially obtain at least one batch of data corresponding to the value of the batch quantity to serve as the target data.

[0016] Optionally, the multiple computing modes include stream processing mode, batch processing mode, and iterative graph computing mode;

[0017] If the target downstream computing node uses a stream processing mode to execute the second subtask, the target data shard contains multiple batches of data, the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data, and the number of batches is infinite.

[0018] If the target downstream computing node uses batch processing mode to execute the second subtask, the target data shard contains a unique batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the unique batch of data, and the batch quantity is 1.

[0019] The batch identifiers of the multiple batches of data are monotonically increasing.

[0020] Optionally, the iterative graph computation mode includes an iterative graph computation mode executed in a stream processing manner; and an iterative graph computation mode executed in a batch processing manner.

[0021] If the target downstream computing node executes the second subtask using an iterative graph computing mode executed in a stream processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration in the iteration; the target data shard contains multiple batches of data; the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data; the number of batches is infinite.

[0022] If the target downstream computing node executes the second subtask using an iterative graph computing mode executed in a batch processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the any iteration; the target data shard contains a unique batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the unique batch of data, and the batch quantity is 1.

[0023] The batch identifiers of the multiple batches of data are monotonically increasing.

[0024] Optionally, the data acquisition conditions include the fragment identifier of the data fragment allocated by the target upstream computing node to the target downstream computing node, and the maximum batch identifier of all data batches to be acquired;

[0025] In response to the data acquisition request, target data that meets the data acquisition conditions is acquired from the data generated during the execution of the first subtask, including:

[0026] In response to the data acquisition request, a target data shard corresponding to the shard identifier is determined from the data shards allocated to the plurality of downstream computing nodes respectively;

[0027] From at least one batch of data contained in the target data segment, at least one batch of data with a batch identifier no greater than the maximum batch identifier is obtained as the target data.

[0028] Optionally, the multiple computing modes include stream processing mode, batch processing mode, and iterative graph computing mode;

[0029] If the target downstream computing node uses a stream processing mode to execute the second subtask, the target data shard contains multiple batches of data, and the maximum batch identifier has a value of infinity.

[0030] If the target downstream computing node uses batch processing mode to execute the second subtask, the target data shard contains data from a unique batch, and the value of the maximum batch identifier is the batch identifier corresponding to the data from the unique batch.

[0031] The batch identifiers of the multiple batches of data are monotonically increasing.

[0032] Optionally, the iterative graph computation mode includes an iterative graph computation mode executed in a stream processing manner; and an iterative graph computation mode executed in a batch processing manner.

[0033] If the target downstream computing node executes the second subtask using an iterative graph computing mode executed in a stream processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the any iteration; the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinity.

[0034] If the target downstream computing node executes the second subtask using an iterative graph computing mode executed in a batch processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the any iteration; the target data shard contains data in a unique batch, and the value of the maximum batch identifier is the batch identifier corresponding to the unique batch of data.

[0035] The batch identifiers of the multiple batches of data are monotonically increasing.

[0036] Optionally, the data acquisition request is an asynchronous request;

[0037] In response to the data acquisition request, target data that meets the data acquisition conditions is acquired from the data generated during the execution of the first subtask, including:

[0038] The system asynchronously responds to the data acquisition request and retrieves target data that meets the data acquisition conditions from the data generated by executing the first subtask.

[0039] Optionally, the target data includes at least one batch of data obtained from data shards allocated to the target downstream computing node;

[0040] Sending the target data to the target downstream computing node includes:

[0041] The data blocks contained in the at least one batch of data obtained are distributed to the target downstream computing node in a streaming manner.

[0042] Optionally, the multiple downstream computing nodes used to execute the second subtask may employ different computing modes to execute the second subtask.

[0043] This specification also proposes a data distribution method for distributed graph computing, wherein computing nodes in the distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; wherein both the first subtask and the second subtask are executed concurrently by multiple computing nodes in the distributed system; the method is applied to any target downstream computing node among the multiple downstream computing nodes executing the second subtask, and the method includes:

[0044] The data acquisition conditions are determined based on the computing mode used to execute the second subtask; wherein, the data acquisition conditions are used to obtain the input data required to execute the second subtask from any target upstream computing node among the multiple upstream computing nodes executing the first subtask;

[0045] Generate a data acquisition request containing the data acquisition conditions, and send the data acquisition request to the target upstream computing node, so that the target upstream computing node can acquire target data that meets the data acquisition conditions from the data generated by executing the first subtask;

[0046] The target data is received from the upstream computing node of the target, and the target data is used as input data to continue executing the second subtask.

[0047] Optionally, the data generated from executing the first subtask includes data shards allocated to multiple downstream computing nodes executing the second subtask; wherein, the data shards contain at least one batch of data; any batch of data contains at least one data block; the number of batches of data contained in the data shard allocated to any downstream computing node is the number corresponding to the computing mode adopted by the downstream computing node in executing the second subtask.

[0048] Optionally, the multiple computing modes include stream processing mode, batch processing mode, and iterative graph computing mode;

[0049] The data acquisition conditions include the fragment identifier of the data fragment allocated by the upstream computing node to the downstream computing node, the batch identifier of the starting data batch to be acquired, and the batch number of the data batch to be acquired.

[0050] If the target downstream computing node uses a stream processing mode to execute the second subtask, the target data shard contains multiple batches of data, the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data, and the number of batches is infinite.

[0051] If the target downstream computing node uses batch processing mode to execute the second subtask, the target data shard contains a unique batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the unique batch of data, and the batch quantity is 1.

[0052] The batch identifiers of the multiple batches of data are monotonically increasing.

[0053] Optionally, the iterative graph computation mode includes an iterative graph computation mode executed in a stream processing manner; and an iterative graph computation mode executed in a batch processing manner.

[0054] If the target downstream computing node executes the second subtask using an iterative graph computing mode executed in a stream processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration in the iteration; the target data shard contains multiple batches of data; the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data; the number of batches is infinite.

[0055] If the target downstream computing node executes the second subtask using an iterative graph computing mode executed in a batch processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the any iteration; the target data shard contains data in a unique batch, the batch identifier of the starting data batch is the batch identifier corresponding to the data in the unique batch, and the batch quantity is 1.

[0056] The batch identifiers of the multiple batches of data are monotonically increasing.

[0057] Optionally, the multiple computing modes include stream processing mode, batch processing mode, and iterative graph computing mode;

[0058] The data acquisition conditions include the fragment identifier of the data fragment allocated by the target upstream computing node to the target downstream computing node, and the maximum batch identifier of all data batches to be acquired;

[0059] If the target downstream computing node uses a stream processing mode to execute the second subtask, the target data shard contains multiple batches of data, and the maximum batch identifier has a value of infinity.

[0060] If the target downstream computing node uses batch processing mode to execute the second subtask, the target data shard contains data from a unique batch, and the value of the maximum batch identifier is the batch identifier corresponding to the data from the unique batch.

[0061] The batch identifiers of the multiple batches of data are monotonically increasing.

[0062] Optionally, the iterative graph computation mode includes an iterative graph computation mode executed in a stream processing manner; and an iterative graph computation mode executed in a batch processing manner.

[0063] If the target downstream computing node executes the second subtask using an iterative graph computing mode executed in a stream processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the any iteration; the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinity.

[0064] If the target downstream computing node executes the second subtask using an iterative graph computing mode executed in a batch processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the any iteration; the target data shard contains data in a unique batch, and the value of the maximum batch identifier is the batch identifier corresponding to the unique batch of data.

[0065] The batch identifiers of the multiple batches of data are monotonically increasing.

[0066] Optionally, the data acquisition request is an asynchronous request.

[0067] Optionally, the target data includes at least one batch of data obtained from a data shard allocated by the upstream computing node to the downstream computing node.

[0068] Receiving the target data sent by the upstream computing node of the target includes:

[0069] Receive data blocks contained in the at least one batch of data sent by the target upstream computing node in a streaming manner.

[0070] Optionally, the multiple downstream computing nodes used to execute the second subtask may employ different computing modes to execute the second subtask.

[0071] In the above embodiments, a unified data distribution method can be provided for the various computing modes supported by the distributed system performing distributed graph computing. For different computing modes, there is no need to provide different data distribution methods separately, thereby reducing the complexity of system development and maintenance and allowing the system to adapt to different computing modes more flexibly. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram illustrating data distribution from an upstream task to a downstream task under different computing modes in one embodiment of this specification;

[0074] Figure 2 This is a flowchart illustrating a data distribution method for distributed graph computing in one embodiment of this specification;

[0075] Figure 3 This is a schematic diagram illustrating an upstream task and a downstream task in one embodiment of this specification;

[0076] Figure 4 This is a schematic diagram illustrating concurrent execution of upstream and downstream tasks in one embodiment of this specification;

[0077] Figure 5 This is a schematic diagram illustrating concurrent execution of upstream and downstream tasks in one embodiment of this specification;

[0078] Figure 6 This is a data structure diagram of the data generated by the upstream computing node executing the first subtask, as shown in one embodiment of this specification;

[0079] Figure 7 This is a flowchart illustrating another data distribution method for distributed graph computing in one embodiment of this specification;

[0080] Figure 8 This is a schematic structural diagram of an electronic device shown in one embodiment of this specification;

[0081] Figure 9 This is a block diagram of a data distribution device for distributed graph computing shown in one embodiment of this specification;

[0082] Figure 10 This is a block diagram of a data distribution apparatus for distributed graph computing, shown in one embodiment of this specification. Detailed Implementation

[0083] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0084] In distributed graph computing scenarios, graph computing engines that support multiple computing modes typically employ different data distribution methods depending on the computing mode. If data distribution (shuffle) is required between upstream and downstream tasks in the distributed graph computing process being executed, different data distribution methods will be used.

[0085] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating how an upstream task distributes data to a downstream task under different computing modes, as shown in this specification.

[0086] like Figure 1 As shown, if the graph computing engine uses batch processing, because batch processing handles large amounts of data and typically has low latency requirements, upstream tasks can usually store the generated batch data in local or external storage after execution. When downstream tasks start, they can send requests to upstream tasks to actively pull batch data from the aforementioned local or external storage as input data. This data distribution method is commonly referred to as Pull-based Shuffle, meaning that downstream tasks actively pull the large amounts of data required by upstream tasks.

[0087] like Figure 1 As shown, if the graph computing engine uses a stream processing model, because stream processing has high requirements for data latency and processes small amounts of data, in this computing model, after the upstream task finishes execution, it can usually divide the generated data into several small batches and then actively push them to the downstream task one batch at a time. This data distribution method is usually called Push-based Shuffle, meaning that the upstream task actively pushes small batches of data to the downstream task.

[0088] like Figure 1 As shown, if the graph computing engine adopts a graph computing mode, due to the relatively unique nature of graph computing, after a computing node completes one iteration of graph computing, it will continue to send the generated data back to itself as input for the next iteration of graph computing. Based on this characteristic, in graph computing mode, data can be distributed using either a pull-based shuffle or a push-based shuffle method.

[0089] For example, please continue to see Figure 1 For graph computation scenarios involving small amounts of data, iterative graph computation can be performed using stream processing, with data from the previous iteration distributed to the next iteration as input using a push-based shuffle approach. For graph computation scenarios involving large amounts of data, iterative graph computation can be performed using batch processing, with data from the previous iteration distributed to the next iteration as input using a pull-based shuffle approach.

[0090] Under different computing modes, the shuffle methods used by upstream and downstream tasks when shuffling data usually differ. This means that graph computing engines need to provide different data shuffle mechanisms for different computing modes.

[0091] It is evident that while distributed systems support various computation modes for distributed graph computation, the different data distribution methods used between upstream and downstream tasks necessitate that the graph computation engine provide different data shuffle mechanisms for each mode. This can lead to increased system complexity and excessive overhead. Furthermore, it may prevent the graph computation engine from flexibly adapting to different computation modes.

[0092] Based on this, this specification proposes a technical solution that provides a unified data distribution method for various computing modes supported by distributed systems performing distributed graph computing.

[0093] Please see Figure 2 , Figure 2 This document presents a flowchart illustrating a data distribution method for distributed graph computing. The computing nodes in the distributed system support executing distributed graph computing tasks according to multiple computing modes. Each distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task. Both the first and second subtasks are executed concurrently by multiple computing nodes in the distributed system. The method is applied to any one of the multiple upstream computing nodes executing the first subtask. The method includes the following execution process:

[0094] Step 202: Receive a data acquisition request from any target downstream computing node among a plurality of downstream computing nodes used to execute the second subtask; wherein, the data acquisition request includes data acquisition conditions determined by the target downstream computing node based on the computing mode adopted to execute the second subtask; the data acquisition conditions are used to obtain the input data required by the target downstream computing node to execute the second subtask from the data generated by the target upstream computing node executing the first subtask; different computing modes adopted by the target downstream computing node to execute the second subtask correspond to different data acquisition conditions;

[0095] The computing nodes in the aforementioned distributed system can specifically support the execution of distributed graph computing tasks according to various computing modes.

[0096] For example, in practical applications, a graph computing engine that supports executing distributed graph computing tasks according to multiple computing modes can be deployed on the computing nodes in a distributed system, so that the computing nodes in the distributed system can also have the ability to execute distributed graph computing tasks according to multiple computing modes.

[0097] It should be noted that, in practical applications, the computing node in the above distributed system can be either an independent node device in a computing cluster or a thread running on a node device to perform graph computing tasks.

[0098] The aforementioned distributed graph computing task can specifically include a first subtask as an upstream task and a second subtask as a downstream task.

[0099] In practical applications, the aforementioned distributed graph computing tasks typically contain multiple subtasks with data dependencies. If the data generated after the completion of a target subtask (i.e., the output data of that subtask) continues to serve as the input data of another subtask, then that target subtask can usually be considered an upstream task, and the other subtask can usually be considered a downstream task corresponding to that upstream task.

[0100] It should be noted that the specific types of subtasks included in the above-mentioned distributed graph computing tasks as upstream and downstream tasks are not specifically limited in this specification. In practical applications, the types of subtasks as upstream and downstream tasks can be the same or different.

[0101] For example, see Figure 3 In one example, a subtask of an upstream task could be a map task within a graph computation task; a subtask of a downstream task could be a sink task within a graph computation task. In another example, a subtask of an upstream task could be a source task within a graph computation task; a subtask of a downstream task could be an iterative graph task within a graph computation task. Alternatively, a subtask of an upstream task could be an iterative graph task within a graph computation task; a subtask of a downstream task could be a sink task within a graph computation task.

[0102] In some embodiments, in distributed graph computing applications, to fully utilize the computing resources in the distributed system, both the first subtask and the second subtask can be executed concurrently by multiple computing nodes in the distributed system. That is, both upstream and downstream tasks can be tasks with a certain degree of concurrency.

[0103] It should be noted that in practical applications, the concurrency levels of upstream and downstream tasks can be the same or different.

[0104] For example, see Figure 4 , Figure 4 This is a schematic diagram illustrating the concurrent execution of upstream and downstream tasks as shown in this specification.

[0105] like Figure 4 As shown, taking the example that upstream and downstream tasks can have the same concurrency, the upstream task can be executed concurrently by three upstream computing nodes (nodes 1-3) in the distributed system, and the downstream task can also be executed concurrently by three downstream computing nodes (nodes AC) in the distributed system.

[0106] In some embodiments, the data generated by the execution of the first subtask by the multiple upstream computing nodes that concurrently execute the first subtask will be used as input data and distributed to the multiple downstream computing nodes that concurrently execute the second subtask. The multiple downstream computing nodes will then continue to execute the second subtask in parallel based on the data distributed by the upstream computing nodes.

[0107] In the context of distributed graph computing, for any one of the multiple upstream computing nodes, the data generated by executing the first subtask can be partitioned according to the concurrency of the downstream computing nodes executing the second subtask. This partitions the data generated by executing the first subtask into multiple data shards that correspond one-to-one with the multiple downstream computing nodes executing the second subtask. These multiple data shards are then distributed to each of the multiple downstream computing nodes.

[0108] For example, see Figure 5 , Figure 5 This is another schematic diagram illustrating the concurrent execution of upstream and downstream tasks as shown in this specification.

[0109] like Figure 5As shown, taking the example of the upstream task being executed concurrently by three upstream computing nodes (nodes 1-3) in the distributed system, and the downstream task being executed concurrently by three downstream computing nodes (nodes A and C) in the distributed system, the upstream node 1 can divide the data generated by executing the first subtask into slice1-A, slice1-B, and slice1-C according to the concurrency degree of the downstream computing nodes (3). Slice1-A is then distributed to downstream node A, slice1-B to downstream node B, and slice1-C to downstream node C. Similarly, upstream node 2 can also divide the data generated from executing the first subtask into slice2-A, slice2-B, and slice2-C according to the concurrency level 3 of the downstream computing nodes, and distribute slice2-A to downstream node A, slice2-B to downstream node B, and slice2-C to downstream node C; upstream node 2 can also divide the data generated from executing the first subtask into slice3-A, slice3-B, and slice3-C according to the concurrency level 3 of the downstream computing nodes, and distribute slice3-A to downstream node A, slice3-B to downstream node B, and slice3-C to downstream node C. Ultimately, the input data for downstream node A when executing the second subtask are slice1-A, slice2-A, and slice3-A; the input data for downstream node B when executing the second subtask are slice1-B, slice2-B, and slice3-B; and the input data for downstream node C when executing the second subtask are slice1-C, slice2-C, and slice3-C.

[0110] In some embodiments, each upstream computing node executing the first subtask can, in addition to dividing the output data generated by executing the first subtask into multiple data shards corresponding one-to-one with the multiple downstream computing nodes executing the second subtask according to the concurrency of the downstream computing nodes, also adopt a unified data structure to abstract each data shard into the form of a data stream.

[0111] It should be noted that, in one scenario, the upstream computing node can, after completing the first subtask, adopt a unified data structure to further abstract each data fragment from the output data generated by the first subtask into a data stream. In another scenario, the upstream computing node can, after completing the first subtask, simply divide the output data generated by the first subtask into multiple data fragments corresponding one-to-one with the multiple downstream computing nodes executing the second subtask, and then, upon receiving a data acquisition request from a downstream computing node, adopt a unified data structure to further abstract each data fragment from the output data generated by the first subtask into a data stream.

[0112] In some embodiments, based on the unified data structure described above, after abstracting each data shard into a data stream, each data shard can specifically contain at least one batch of data, and any batch of data can specifically contain at least one data block. For example, please refer to... Figure 6 , Figure 6 This is a data structure diagram of the data generated by an upstream computing node executing the first subtask, as shown in this specification.

[0113] like Figure 6 As shown, the data structure of the data generated by the upstream computing node executing the first subtask can specifically include Slice, batch, and Message.

[0114] The Slice mentioned above is the basic unit for managing the data generated by executing the first subtask. Typically, upstream computing nodes divide the data generated by executing the second subtask into N slices, based on the concurrency N of downstream computing nodes executing the upstream task. Each slice can also be further divided into... Figure 5 The unified data structure shown can be further abstracted into the form of data streams.

[0115] The aforementioned "Batch" refers to the basic unit of data management within a Slice. A Batch represents a batch of data, and a Slice can contain one or more batches of data. In other words, a Slice can specifically be a data stream composed of one or more batches of data. Within a Slice, different batches can be separated using barriers. A barrier can be understood as a separator between different batches, used to distinguish them and indicate the end position of the current batch. Within a barrier, metadata about the current batch can be recorded; for example, this metadata may include batchID, batch length, and other information.

[0116] It should be noted that the number of data batches contained within a slice that an upstream computing node allocates for any downstream computing node is typically a number corresponding to the computing mode used by that downstream computing node to execute the downstream task. In other words, the number of data batches contained within a slice that an upstream computing node allocates for a downstream computing node usually depends on the computing mode used by that downstream computing node to execute the downstream task.

[0117] For example, in a distributed graph computing scenario, the aforementioned computing modes can specifically include stream processing mode, batch processing mode, and iterative graph computing mode.

[0118] In stream processing mode, a slice can contain multiple batches of data. In batch processing mode, a slice can contain only one batch of data. For iterative graph processing, in practice, it can be executed either as stream processing or as batch processing. On the one hand, if the iterative graph computation is executed as stream processing, a slice can contain multiple batches of data; on the other hand, if the iterative graph computation is executed as batch processing, a slice can contain only one batch of data.

[0119] In practical applications, after an upstream computing node allocates a slice for any downstream computing node, it can further obtain the computing mode adopted by the downstream computing node when executing the second subtask. Then, based on the obtained computing mode, it can determine the number of data batches that the slice allocated for the downstream computing node should contain, and based on the determined number, further divide the data contained in the slice into one or more batches of data. In this way, the data contained in the slice allocated for the downstream computing node is further abstracted into a data flow.

[0120] For example, in one scenario, if the upstream computing node, after completing the first subtask, adopts the unified data structure described above to further abstract each data slice from the output data generated by the first subtask into a data stream, then the computing mode used by the downstream computing node to execute the second subtask can be obtained from the task scheduler (this computing mode is usually planned in advance when specifying the execution plan of the graph computing task). Based on the obtained computing mode, the number of data batches that should be contained within the slice divided by the downstream computing node can be determined, and based on the determined number, the data contained within the slice can be further divided into one or more batches of data.

[0121] In another scenario, if after completing the first subtask, the upstream computing node simply divides the output data generated by the first subtask into multiple data slices corresponding one-to-one with the multiple downstream computing nodes executing the second subtask, and then, upon receiving a data acquisition request from a downstream computing node, uses a unified data structure to further abstract each data slice from the output data generated by the first subtask into a data stream, then during the data acquisition phase, the computing mode used by the downstream computing node to execute the second subtask can be determined based on the data acquisition conditions contained in the data acquisition request sent by the downstream computing node. Then, based on the acquired computing mode, the number of data batches that should be contained within the slice divided by the downstream computing node can be determined, and based on the determined number, the data contained within the slice can be further divided into one or more batches of data.

[0122] The term "Message" refers to the basic management unit of data contained within a batch. A Message can represent a smaller data block obtained by dividing the data contained in a batch. The data in a single batch can be further divided into one or more Messages; that is, a batch can specifically be a data stream composed of one or more Messages. Besides recording some of the data contained in a batch, a Message can also record some metadata of the current batch in practical applications, such as batchID and batch length. It should be noted that the number of Messages ultimately divided from the data contained in a batch usually depends on the total length of the data contained in the batch and the unit length set for each Message. In practical applications, the unit length can be flexibly set for Messages based on specific needs to divide the data contained in a batch.

[0123] In some embodiments, the data distribution methods provided for the above-mentioned multiple computing modes in the distributed system can be unified into a Pull-based Shuffle data distribution method; that is, the downstream computing node actively sends a data acquisition request to the upstream computing node, and the upstream computing node can process the data acquisition request and distribute the data required by the downstream computing node to the downstream computing node as needed.

[0124] In this case, for any target downstream computing node among the multiple downstream computing nodes used to execute the second subtask, the data acquisition conditions corresponding to the computing mode can be determined first based on the computing mode adopted to execute the second subtask.

[0125] Specifically, the data acquisition condition is used to obtain the input data required by the target downstream computing node to execute the second subtask from the data generated by each upstream computing node executing the first subtask. Different computing modes employed by the target downstream computing node to execute the second subtask can correspond to different data acquisition conditions; that is, the data acquisition condition can differ under different computing modes. The upstream computing node should be able to accurately determine the computing mode employed by the downstream computing node to execute the second subtask based on the differences in the data acquisition conditions contained in the data acquisition requests sent by the downstream computing node.

[0126] Once the downstream computing node determines the data acquisition conditions corresponding to the computing mode it uses to execute the second subtask, it can construct data acquisition requests for the multiple upstream computing nodes used to execute the first subtask based on the determined data acquisition conditions. In other words, multiple data acquisition requests are constructed based on the concurrency of the upstream computing nodes executing the first subtask. For example, if the concurrency of the upstream computing nodes executing the first subtask is N, then N data acquisition requests need to be constructed.

[0127] Then, the constructed data acquisition requests can be sent to the aforementioned upstream computing nodes respectively.

[0128] In some embodiments, under different computing modes, the upstream computing node executing the first subtask and the downstream computing node executing the second subtask can run simultaneously or sequentially.

[0129] On the one hand, if the upstream computing node executing the first subtask and the downstream computing node executing the second subtask are running simultaneously, the downstream computing node executing the second subtask can immediately send a data acquisition request to the upstream computing node after the upstream computing node and the downstream computing node start up simultaneously.

[0130] On the other hand, if the upstream computing node executing the first subtask runs before the downstream computing node executing the second subtask, the downstream computing node executing the second subtask can also send a data acquisition request to the upstream computing node after the upstream computing node has completed executing the second subtask.

[0131] For example, in batch processing mode, downstream computing nodes are usually scheduled to continue executing the second subtask only after the upstream computing node has completed the first subtask. Therefore, based on this task execution characteristic in batch processing mode, the upstream computing node usually starts before the downstream computing node. The downstream computing node will only start after the upstream computing node has completed the first subtask. In this case, the downstream computing node can send a data acquisition request to the upstream computing node after the upstream computing node has completed the first subtask and started itself.

[0132] In another example, in stream processing mode, upstream and downstream compute nodes are typically started simultaneously. In this case, the downstream compute node can send a data retrieval request to the upstream compute node immediately after both upstream and downstream compute nodes are started.

[0133] In the third example, under the iterative graph computation model, since the upstream and downstream compute nodes typically correspond to the same compute node, they can be two different threads running on the same compute node, and these two threads can usually start simultaneously. In this case, the thread corresponding to the downstream compute node can start simultaneously with both the thread corresponding to the upstream compute node and the thread corresponding to the downstream compute node, and then immediately send a data retrieval request to the thread corresponding to the upstream compute node.

[0134] Step 204: In response to the data acquisition request, obtain target data that meets the data acquisition conditions from the data generated by executing the first subtask;

[0135] For any target upstream computing node among the multiple upstream computing nodes executing the first subtask described above, it can receive a data acquisition request sent by a downstream computing node. After receiving a data acquisition request sent by any target downstream computing node among the multiple downstream computing nodes, it can further respond to the data acquisition request and distribute data to the target downstream computing node using a unified Pull-based Shuffle data distribution method for different computing modes.

[0136] It should be noted that, since a unified Pull-based Shuffle data distribution method can be used to distribute data to downstream computing nodes for different computing modes, the upstream computing nodes can use the same data distribution method to distribute data to downstream computing nodes regardless of the computing mode used by the downstream computing nodes. Therefore, in this case, multiple downstream computing nodes used to execute the second subtask can use different computing modes to execute the second subtask in parallel.

[0137] In some embodiments, the unified pull-based shuffle data distribution method provided for the above-mentioned multiple computing modes can specifically be a non-blocking pull mode.

[0138] In this non-blocking pull mode, the data retrieval request sent by the downstream computing node to the upstream computing node can be an asynchronous request. Correspondingly, after receiving the data retrieval request from the downstream computing node, the upstream computing node can asynchronously respond to the request and retrieve the target data that meets the data retrieval conditions from the data generated during the execution of the first subtask.

[0139] It should be noted that the specific form of the data acquisition conditions included in the above data acquisition request is not specifically limited in this specification. In practical applications, the above data acquisition conditions can be obtained by explicitly carrying the fragment identifier of the data fragment allocated to the downstream computing node and the batch identifier or batch number of the data batch to be acquired in the above data acquisition request. Alternatively, the above data acquisition conditions can be obtained by carrying some implicit data acquisition rules in the above data acquisition request. These will not be listed one by one in this specification.

[0140] In some embodiments, the data acquisition conditions included in the above data acquisition request may specifically include the fragment identifier of the data fragment allocated by the target upstream computing node to the target downstream computing node, the batch identifier of the starting data batch to be acquired, and the batch number of the data batch to be acquired.

[0141] In this scenario, after receiving a data acquisition request from the downstream computing node, the upstream computing node can first acquire the fragment identifier, batch identifier, and batch quantity contained in the data acquisition request as data acquisition conditions.

[0142] Secondly, the target data fragment corresponding to the fragment identifier can be determined from the data fragments allocated to the multiple downstream computing nodes mentioned above.

[0143] For example, in practical applications, when an upstream computing node allocates data shards to a downstream computing node, it can, by default, use the downstream computing node's node identifier as the shard identifier, or it can use the downstream computing node's node identifier as part of the shard identifier. Thus, when the target downstream computing node constructs a data retrieval request, it can, by default, include its own node identifier as the shard identifier of the data shard allocated to it by the target upstream computing node in the data retrieval request. Upon receiving the data retrieval request, the target upstream computing node can directly match the shard identifier contained in the request with the shard identifiers already allocated to each downstream computing node to determine the target data shard corresponding to the shard identifier contained in the data retrieval request.

[0144] Then, the target data batch corresponding to the aforementioned batch identifier can be further determined from at least one batch of data contained in the target data slice.

[0145] For example, in practical applications, the batch identifier of each batch of data included in the data shard allocated by the upstream computing node to the downstream computing node can be a monotonically increasing identifier starting from a default value (such as 0 or 1). In this case, the batch identifier of the starting data batch included in the data acquisition request sent by the target downstream computing node to the target upstream computing node can be this default value, which represents the batch identifier corresponding to the first batch of data included in the target data shard. After receiving the data acquisition request, the target upstream computing node can, by default, determine the first batch of data included in the target data shard as the target data batch corresponding to the aforementioned batch identifier.

[0146] Finally, the target data batch can be used as the starting data batch, and at least one batch of data corresponding to the above batch number can be obtained in sequence as the target data; for example, if the above batch number is N, then it is necessary to start from the target data batch and obtain N batches of data in sequence as the target data; then, the obtained target data is sent to the target downstream computing node.

[0147] For example, taking the aforementioned various computing modes, including stream processing, batch processing, and iterative graph computing, as an example, if the downstream computing node uses stream processing to execute the second subtask, the target data shard allocated by the upstream computing node to the downstream computing node can specifically contain multiple batches of data. In this case, the batch identifier of the starting data batch included in the data acquisition request sent by the downstream computing node to the upstream computing node can specifically be the batch identifier corresponding to the first batch of data among the multiple batches of data contained in the target data shard. For example, the batch identifier of the multiple batches of data contained in the target data shard can specifically be an identifier that monotonically increases from a default value as the starting value, and the identifier of the first batch of data can be a default value as the starting value. The number of batches can be infinite (for example, it can be a character representing infinity), indicating that all batches of data contained in the target data shard should be acquired.

[0148] If the target downstream computing node uses batch processing mode to execute the second subtask, the target data shard allocated by the target upstream computing node to the target downstream computing node may contain only one batch of data. In this case, the batch identifier of the starting data batch included in the data acquisition request sent by the target downstream computing node to the target upstream computing node may be the batch identifier corresponding to the only batch of data. For example, the batch identifier corresponding to the only batch of data may be a default value (such as 0 or 1) as the starting value. The value of the batch number may be 1, indicating that in batch processing mode, only the only batch of data contained in the target data shard needs to be acquired.

[0149] If the downstream computing node of the aforementioned target executes the second subtask using an iterative graph processing mode, meaning that the second subtask itself is an iterative graph computation subtask, then since iterative graph processing modes typically include iterative graph computation modes executed in a stream processing manner and iterative graph computation modes executed in a batch processing manner, these two cases need to be discussed separately:

[0150] On the one hand, if the downstream computing node executes the second subtask using an iterative graph computing mode executed in a stream processing manner, then the first subtask can typically be any iteration within the iterative graph computing, and the second subtask can typically be the next iteration in that iteration. The target data shard allocated by the upstream computing node to the downstream computing node can specifically contain multiple batches of data. In other words, in the iterative graph computing mode executed in a stream processing manner, the data contained in the target data shard will be further divided into multiple batches and distributed to the downstream computing node in multiple batches.

[0151] In this scenario, the batch identifier of the initial data batch included in the data acquisition request sent by the downstream computing node to the upstream computing node can specifically be the batch identifier corresponding to the first batch of data among multiple batches contained in the target data shard. For example, the batch identifier of the multiple batches of data contained in the target data shard can be an identifier that monotonically increases from a default value as the starting value, in which case the identifier of the first batch of data can be a default value as the starting value. The number of batches can be infinite.

[0152] On the other hand, if the downstream computing node executes the second subtask using an iterative graph computing mode executed in batch processing, the first subtask can still be any iteration within the iterative graph computing. The second subtask can typically be the next iteration in that iteration. The target data shard allocated by the upstream computing node to the downstream computing node can specifically contain only one batch of data. In other words, in the iterative graph computing mode executed in batch processing, the data in the target data shard is further divided into unique batches and then distributed to the downstream computing node in batches.

[0153] In this case, the batch identifier of the starting data batch included in the data acquisition request sent by the downstream computing node to the upstream computing node can specifically be the batch identifier corresponding to the unique batch of data; for example, the batch identifier corresponding to the unique batch of data can be a default value (such as 0 or 1) as the starting value; the value of the batch number can be 1, indicating that in the iterative graph computing mode executed in batch processing mode, only the data of the unique batch contained in the target data shard needs to be acquired.

[0154] In some embodiments, the data acquisition conditions included in the above data acquisition request may further include the fragment identifier of the data fragment allocated by the target upstream computing node to the target downstream computing node, and the maximum batch identifier of all data batches to be acquired.

[0155] In this scenario, after receiving a data acquisition request from the target downstream computing node, the upstream computing node can first acquire the aforementioned fragment identifier and the aforementioned maximum batch identifier, which are included in the data acquisition request as data acquisition conditions.

[0156] Secondly, a target data shard corresponding to the aforementioned shard identifier can be determined from the data shards allocated to the aforementioned multiple downstream computing nodes. Then, from at least one batch of data contained in the aforementioned target data shard, at least one batch of data with a batch identifier not greater than the aforementioned maximum batch identifier can be obtained as the aforementioned target data.

[0157] For example, taking the aforementioned various computing modes, including stream processing, batch processing, and iterative graph computing, as an example, if the downstream computing node of the target uses stream processing mode to execute the second subtask, the target data shard can contain multiple batches of data. For instance, the batch identifier of the multiple batches of data contained in the target data shard can be a monotonically increasing identifier starting from a default value (such as 0 or 1). The value of the maximum batch identifier can be infinity, indicating that all batches of data contained in the target data shard should be obtained.

[0158] If the downstream computing node of the target uses batch processing mode to execute the second subtask, the target data shard may contain only one batch of data; for example, the batch identifier of the data pair of the single batch can be a default value (such as 0 or 1) as the starting value; the value of the maximum batch identifier can be the batch identifier corresponding to the data of the single batch.

[0159] If the downstream computing node of the aforementioned target executes the second subtask using an iterative graph processing mode, then the second subtask itself is an iterative graph computation subtask. In this case, since iterative graph processing modes typically include iterative graph computation modes executed in a stream processing manner and iterative graph computation modes executed in a batch processing manner, these two scenarios need to be discussed separately:

[0160] On the one hand, if the downstream computing node of the target uses an iterative graph computing mode that is executed in a stream processing manner to perform the second subtask, then the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the iteration; the target data shard can contain multiple batches of data, and the value of the maximum batch identifier can be infinite, indicating that the number of all batches contained in the target data shard should be obtained;

[0161] On the other hand, if the downstream computing node of the target uses an iterative graph computing mode that is executed in batch processing when performing the second subtask, the first subtask is still a computing task of any iteration in the iterative graph computing, and the second subtask is a computing task of the next iteration of that iteration; the target data shard may contain only one batch of data, and the value of the maximum batch identifier can be the batch identifier corresponding to the data of that single batch.

[0162] Step 206: The target data is sent to the target downstream computing node so that the target downstream computing node can use the target data as input data to continue executing the second subtask.

[0163] After obtaining the target data that meets the aforementioned data acquisition conditions from the data generated during the execution of the first subtask, the upstream node can distribute the target data to the downstream computing node. This target data typically includes at least one batch of data obtained from a data shard allocated to the downstream computing node.

[0164] After receiving the target data from the target upstream node, the downstream computing node can use the target data as input data to continue executing the second subtask.

[0165] It should be noted that, since the multiple upstream computing nodes that execute the first subtask in parallel will each allocate a data shard from the data generated by the execution of the first subtask to the target downstream computing node, the target downstream computing node can obtain the target data from the data shards allocated to it by each of the multiple upstream computing nodes by sending a data acquisition request to the multiple upstream computing nodes. Then, it can integrate the multiple target data obtained from the data shards allocated to it by each of the upstream computing nodes as input data to continue executing the second subtask.

[0166] For example, please continue to see Figure 5 For downstream computing node A, slice1-A generated by upstream computing node 1 for downstream computing node A, slice2-A generated by upstream computing node 2 for downstream computing node A, and slice3-A generated by upstream computing node 3 for downstream computing node A can be obtained respectively. Then, slice1-A, slice2-A and slice3-A are integrated and used as input data to continue to execute the second subtask mentioned above.

[0167] In some embodiments, since a unified pull-based shuffle data distribution method can be used to distribute data to downstream computing nodes for different computing modes, and a unified data structure can be used to further abstract the data fragments allocated to each downstream subtask from the output data generated by the execution of upstream tasks into the form of data streams; therefore, adopting this unified data distribution mode is equivalent to abstracting the batch data that needs to be processed in the batch processing mode into a data stream with a batch of 1, and abstracting the data generated by N iterations that need to be processed in the iterative graph computing mode into a finite number of batch data streams.

[0168] Based on this characteristic, when the upstream computing node of the target sends the acquired target data to the downstream computing node of the target, it can specifically send data blocks from at least one batch of data contained in the acquired target data to the downstream computing node of the target in a streaming manner.

[0169] For example, please continue to see Figure 6 The message in at least one slice of the acquired target data can be sent one by one to the downstream computing node of the target using a streaming method, with each message as the smallest transmission unit.

[0170] In some embodiments, since the upstream target computing node and the downstream target computing node usually correspond to the same computing node in the iterative graph computing mode, the upstream target computing node and the downstream target computing node can usually be two different threads running on the same computing node. Therefore, if the computing mode adopted by the downstream target computing node to execute the second subtask is the iterative graph computing mode, the first subtask is usually the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of that iteration.

[0171] In this case, when the thread corresponding to the upstream computing node of the target sends the acquired target data to the downstream computing node of the target, it can specifically send the target data to a local storage space pre-specified on the computing node where it is located; for example, it can write data blocks from at least one batch of data contained in the acquired target data into the local storage space one by one in a streaming manner.

[0172] The thread corresponding to the downstream computing node of the aforementioned target can specifically read the target data from the local storage space as input data for executing the second subtask.

[0173] The local storage space can be a memory space or a disk storage space pre-specified on the computing node.

[0174] In some embodiments, if the downstream computing node uses a batch processing mode to execute the downstream task, and if the upstream and downstream tasks have the same concurrency, the second subtask executed by the downstream computing node and the first subtask executed by the upstream computing node can be deployed to the same computing node for execution. That is, in a batch processing mode where the upstream and downstream tasks have the same concurrency, the upstream and downstream computing nodes can correspond to the same computing node, and the upstream and downstream computing nodes can be two different threads running on the same computing node.

[0175] In this scenario, since upstream computing nodes typically start before downstream computing nodes in batch processing mode, once the thread corresponding to the aforementioned target upstream computing node has started, an additional independent thread can be started on the computing node where it is located for the thread corresponding to the aforementioned target downstream computing node. This independent thread will run independently of the thread corresponding to the aforementioned target downstream computing node. After this independent thread starts, it can send data acquisition requests to the multiple upstream computing nodes used to execute the aforementioned first subtask in advance.

[0176] In this way, since the thread corresponding to the upstream computing node can receive the data acquisition request sent by the thread corresponding to the downstream computing node in advance, the thread corresponding to the upstream computing node can obtain the target data from the data fragment allocated to it in advance and can pre-download the obtained target data to a pre-specified local storage space on its computing node. Subsequently, when the thread corresponding to the downstream computing node starts, it can directly read the target data from this local storage space as input data for executing the second subtask, without needing to repeatedly send data acquisition requests.

[0177] Please see Figure 7 , Figure 7 This is a flowchart illustrating another data distribution method for distributed graph computing as shown in this specification; the computing nodes in the distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; wherein, both the first subtask and the second subtask are executed concurrently by multiple computing nodes in the distributed system; the method is applied to any target downstream computing node among the multiple downstream computing nodes executing the second subtask; the method includes the following execution process:

[0178] Step 702: Determine data acquisition conditions based on the computing mode used to execute the second subtask; wherein, the data acquisition conditions are used to obtain the input data required to execute the second subtask from any target upstream computing node among the multiple upstream computing nodes executing the first subtask;

[0179] Step 704: Generate a data acquisition request containing the data acquisition conditions, and send the data acquisition request to the target upstream computing node, so that the target upstream computing node can acquire target data that meets the data acquisition conditions from the data generated by executing the first subtask;

[0180] Step 704: Receive the target data sent by the upstream computing node of the target, and use the target data as input data to continue executing the second subtask.

[0181] It should be noted that the above Figure 2 Steps 202-204 in the execution flow shown are a method flow in which any one of the target upstream computing nodes among the multiple downstream computing nodes executing the first subtask (i.e., the upstream task) is used as the execution subject, and is related to... Figure 2 The difference is, Figure 7 Steps 702-704 in the execution flow shown are a method flow that uses any one of the multiple downstream computing nodes executing the second subtask as the execution subject. Because... Figure 7 The implementation details of the method flow shown in the figure are consistent with... Figure 2 The implementation details of the method flow shown in the previous example are exactly the same, therefore, they will not be repeated in this embodiment. Figure 7 The implementation details of each step are not elaborated here, and those skilled in the art can refer to the descriptions in the previous embodiments.

[0182] The above technical solutions can provide a unified data distribution method for the various computing modes supported by the distributed system performing distributed graph computing. For different computing modes, there is no need to provide different data distribution methods separately, which can reduce the complexity of system development and maintenance, and allow the system to adapt to different computing modes more flexibly.

[0183] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of apparatus, electronic devices, and storage media.

[0184] Figure 8 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. Please refer to... Figure 8At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, memory 808, and non-volatile memory 810, and may also include other necessary hardware. One or more embodiments of this specification can be implemented in software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into memory 808 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0185] like Figure 9 As shown, Figure 9 This specification is a block diagram illustrating a data distribution apparatus for distributed graph computing according to an exemplary embodiment, the apparatus being operable in, for example... Figure 8 The illustrated electronic device implements the technical solution of this specification. The computing nodes in the distributed system for performing distributed graph computing tasks support execution of the distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; both the first subtask and the second subtask are executed concurrently by multiple computing nodes in the distributed system; the device is applied to any target upstream computing node among the multiple upstream computing nodes executing the first subtask; the device 90 includes:

[0186] The receiving module 901 receives a data acquisition request sent by any target downstream computing node among a plurality of downstream computing nodes used to execute the second subtask; wherein, the data acquisition request includes data acquisition conditions determined by the target downstream computing node based on the computing mode adopted to execute the second subtask; the data acquisition conditions are used to obtain the input data required by the target downstream computing node to execute the second subtask from the data generated by the target upstream computing node executing the first subtask; different computing modes adopted by the target downstream computing node to execute the second subtask correspond to different data acquisition conditions;

[0187] Acquisition module 902, in response to the data acquisition request, acquires target data that meets the data acquisition conditions from the data generated by executing the first subtask;

[0188] The distribution module 903 distributes the target data to the target downstream computing node, so that the target downstream computing node can use the target data as input data to continue executing the second subtask.

[0189] like Figure 10 As shown, Figure 10 This is a block diagram illustrating another data distribution apparatus for distributed graph computing according to an exemplary embodiment of this specification, which can also operate in, for example... Figure 8 The illustrated electronic device implements the technical solution of this specification. The computing nodes in the distributed system for performing distributed graph computing tasks support execution of the distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; wherein both the first subtask and the second subtask are executed concurrently by multiple computing nodes in the distributed system; the method is applied to any target downstream computing node among the multiple downstream computing nodes executing the second subtask; the device 100 includes:

[0190] The determining module 1001 determines data acquisition conditions based on the computing mode used to execute the second subtask; wherein, the data acquisition conditions are used to obtain the input data required to execute the second subtask from any target upstream computing node among the multiple upstream computing nodes executing the first subtask;

[0191] The sending module 1002 generates a data acquisition request containing the data acquisition conditions and sends the data acquisition request to the target upstream computing node, so that the target upstream computing node can acquire target data that meets the data acquisition conditions from the data generated by executing the first subtask;

[0192] The execution module 1003 receives the target data sent by the upstream computing node of the target and uses the target data as input data to continue executing the second subtask.

[0193] Accordingly, this specification also provides an electronic device including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement all the steps in the previously described method flow.

[0194] Accordingly, this specification also provides a computer-readable storage medium having stored thereon executable computer program instructions; wherein, when executed by a processor, the instructions implement all the steps in the previously described method flow.

[0195] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, they implement all the steps in the previously described method flow.

[0196] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0197] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0198] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0199] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0200] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0201] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0204] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0205] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0206] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0207] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0209] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0210] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A data distribution method for distributed graph computing, wherein computing nodes in a distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; The method is applied to an upstream computing node executing the first subtask, and the method includes: The system receives a data acquisition request sent by a downstream computing node for executing the second subtask; wherein the data acquisition request includes data acquisition conditions determined by the downstream computing node based on the computing mode adopted for executing the second subtask; the data acquisition conditions are used to obtain the input data required by the downstream computing node for executing the second subtask from the data generated by the upstream computing node executing the first subtask; different computing modes adopted by the downstream computing node for executing the second subtask correspond to different data acquisition conditions. In response to the data acquisition request, target data that meets the data acquisition conditions is obtained from the data generated by executing the first subtask; The target data is sent to the downstream computing node, so that the downstream computing node can use the target data as input data to continue executing the second subtask.

2. The method as described in claim 1, wherein the first subtask and the second subtask are both executed concurrently by multiple computing nodes in the distributed system; the data generated by executing the first subtask includes data shards allocated to the multiple downstream computing nodes executing the second subtask respectively; the data shards include at least one batch of data; any batch of data includes at least one data block; in, The number of data batches contained in the data shard allocated to any downstream computing node is the number corresponding to the computing mode used by that downstream computing node to execute the second subtask.

3. The method as described in claim 2, wherein the data acquisition conditions include the fragment identifier of the data fragment allocated by the upstream computing node to the downstream computing node, the batch identifier of the starting data batch to be acquired, and the batch number of the data batch to be acquired; In response to the data acquisition request, target data that meets the data acquisition conditions is acquired from the data generated during the execution of the first subtask, including: In response to the data acquisition request, a target data shard corresponding to the shard identifier is determined from the data shards allocated to the plurality of downstream computing nodes respectively; From at least one batch of data contained in the target data segment, determine the target data batch corresponding to the batch identifier, and use the target data batch as the starting data batch. Sequentially obtain at least one batch of data corresponding to the value of the batch quantity to serve as the target data.

4. The method as described in claim 3, wherein the multiple computing modes include stream processing mode, batch processing mode, and iterative graph computing mode; If the downstream computing node uses a streaming processing mode to execute the second subtask, the target data shard contains multiple batches of data, the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data, and the number of batches is infinite. If the downstream computing node uses batch processing mode to execute the second subtask, the target data shard contains a unique batch of data, the batch identifier of the starting data batch is the batch identifier corresponding to the unique batch of data, and the batch quantity is 1. in, The batch identifiers of the data from the multiple batches are monotonically increasing.

5. The method of claim 4, wherein the iterative graph computation mode includes an iterative graph computation mode executed in a stream processing manner; and an iterative graph computation mode executed in a batch processing manner; If the downstream computing node executes the second subtask using an iterative graph computing mode executed in a stream processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration in the iteration; the target data shard contains multiple batches of data; the batch identifier of the starting data batch is the batch identifier corresponding to the first batch of data among the multiple batches of data; the number of batches is infinite. If the downstream computing node executes the second subtask using an iterative graph computing mode executed in a batch manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the iteration; the target data shard contains data in a unique batch, the batch identifier of the starting data batch is the batch identifier corresponding to the data in the unique batch, and the batch quantity is 1. in, The batch identifiers of the data from the multiple batches are monotonically increasing.

6. The method as described in claim 2, wherein the data acquisition conditions include the fragment identifier of the data fragment allocated by the upstream computing node to the downstream computing node, and the maximum batch identifier of all data batches to be acquired; In response to the data acquisition request, target data that meets the data acquisition conditions is acquired from the data generated during the execution of the first subtask, including: In response to the data acquisition request, a target data shard corresponding to the shard identifier is determined from the data shards allocated to the plurality of downstream computing nodes respectively; From at least one batch of data contained in the target data segment, at least one batch of data with a batch identifier no greater than the maximum batch identifier is obtained as the target data.

7. The method as described in claim 6, wherein the multiple computing modes include stream processing mode, batch processing mode, and iterative graph computing mode; If the downstream computing node uses a streaming processing mode to execute the second subtask, the target data shard contains multiple batches of data, and the maximum batch identifier is infinitely large. If the downstream computing node uses batch processing mode to execute the second subtask, the target data shard contains data from a unique batch, and the value of the maximum batch identifier is the batch identifier corresponding to the data from the unique batch. in, The batch identifiers of the data from the multiple batches are monotonically increasing.

8. The method of claim 7, wherein the iterative graph computation mode includes an iterative graph computation mode executed in a stream processing manner; and an iterative graph computation mode executed in a batch processing manner; If the downstream computing node executes the second subtask using an iterative graph computing mode executed in a stream processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the any iteration; the target data shard contains multiple batches of data, and the value of the maximum batch identifier is infinity. If the downstream computing node executes the second subtask using an iterative graph computing mode executed in a batch processing manner, the first subtask is the computing task of any iteration in the iterative graph computing, and the second subtask is the computing task of the next iteration of the any iteration; the target data shard contains data in a unique batch, and the value of the maximum batch identifier is the batch identifier corresponding to the data in the unique batch. in, The batch identifiers of the data from the multiple batches are monotonically increasing.

9. The method as described in claim 2, wherein the data acquisition request is an asynchronous request; In response to the data acquisition request, target data that meets the data acquisition conditions is acquired from the data generated during the execution of the first subtask, including: The system asynchronously responds to the data acquisition request and retrieves target data that meets the data acquisition conditions from the data generated by executing the first subtask.

10. The method of claim 9, wherein the target data comprises at least one batch of data obtained from a data shard allocated to the downstream computing node; Sending the target data to the downstream computing nodes includes: The data blocks contained in the at least one batch of data are streamed to the downstream computing nodes.

11. The method of claim 1, wherein multiple downstream computing nodes for executing the second subtask employ different computing modes to execute the second subtask.

12. A data distribution method for distributed graph computing, wherein computing nodes in a distributed system support executing distributed graph computing tasks according to multiple computing modes; the distributed graph computing task includes a first subtask as an upstream task and a second subtask as a downstream task; The method is applied to a downstream computing node executing the second subtask, and the method includes: The data acquisition conditions are determined based on the computation mode used to execute the second subtask; wherein, the data acquisition conditions are used to obtain the input data required to execute the second subtask from the upstream computing node executing the first subtask; A data acquisition request containing the data acquisition conditions is generated and sent to the upstream computing node so that the upstream computing node can acquire target data that meets the data acquisition conditions from the data generated by executing the first subtask; The target data sent by the upstream computing node is received, and the target data is used as input data to continue executing the second subtask.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 12.

15. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method of any one of claims 1 to 12.

Images