Graph calculation method, graph calculation device and graph calculation engine based on asynchronous parallel processing

Abstract

The embodiment of the invention provides a graph calculation method, a graph calculation device and a graph calculation engine based on asynchronous parallel processing. When the first step operator is scheduled by the scheduler from the step operator set to the first computing node and one or more second computing nodes for parallel execution, responding to message data which is received from other computing nodes and is required by the first step operator for executing the current iterative computation; the first computing node executes the current iterative computation of the first step operator by using the received message data, and asynchronously sends the current iterative computation result of the first step operator as the message data to the computing node to which the downstream step operator of the first step operator is scheduled; and waiting for the second computing node to complete the current iterative computation of the operator in the first step is not needed.

Description

Technical Field

[0001] The embodiments of this specification generally relate to the field of graph computing, and in particular to a graph computing method, a graph computing device, and a graph computing engine based on asynchronous parallel processing (ASP). Background Art

[0002] In reality, there are many kinds of relationship networks, such as interpersonal relationship networks and transaction capital networks. These relationship network data can be abstracted into a graph data structure consisting of points and edges, which contains multiple attributes. A real-time graph computing engine can be used to perform graph computing based on graph data to mine group characteristics in complex relationship networks. For example, these relationship networks can be matched and traversed to mine valuable information.

[0003] In graph traversal scenarios with large requests / full traversals (for example, online financial risk control scenarios), there will be graph computing tasks involving iterative computing. In this case, using the Bulk Synchronous Parallel (BSP) model for graph computing will result in message backlogs, greatly increasing memory pressure. Summary of the invention

[0004] The embodiments of this specification provide a graph computing solution based on asynchronous parallel processing. Using this graph computing solution, when performing iterative calculations, after the computing nodes participating in the current iterative calculations generate message data, they are immediately sent to the computing nodes participating in the next round of iterative calculations for rapid processing, without waiting for other computing nodes participating in the current iterative calculations to complete the iterative calculations. This eliminates the barrier dependency between iterations that occurs when performing iterative calculations based on the BSP model, reduces message backlogs, and implements traversal iterations in a pipelined manner.

[0005] According to one aspect of an embodiment of the present specification, a graph computing method based on asynchronous parallel processing is provided, the graph computing method being executed by a first computing node in a computing node cluster, the graph computing method comprising: in response to receiving message data required for a first step operator to execute a current iterative calculation from other computing nodes in the computing node cluster, using the received message data to execute the current iterative calculation of the first step operator, the first step operator being scheduled by a scheduler from a step operator set to the first computing node and one or more second computing nodes for parallel execution, the step operator set being obtained by expanding an executable grouped loop for iterative calculation in a graph logic execution plan; and asynchronously sending the current iterative calculation result of the first step operator as message data to the computing node to which the downstream step operator of the first step operator is scheduled, without waiting for all the second computing nodes to complete the current iterative calculation of the first step operator.

[0006] In an example of the above aspect, the current iterative calculation result of the first step operator is asynchronously sent to the computing node to which the downstream step operator of the first step operator is scheduled in the form of shuffle data.

[0007] In an example of the above aspect, asynchronously sending the current iterative calculation result of the first step operator to the computing node to which the downstream step operator of the first step operator is scheduled includes: asynchronously sending the current iterative calculation result of the first step operator to the computing node to which the downstream step operator of the first step operator is scheduled in a message window of a specified size.

[0008] In one example of the above aspect, message data within the same executable group is exchanged in a pipeline mode, and message data between different executable groups is exchanged in a batch mode.

[0009] In an example of the above aspect, message data of other computing nodes in the computing node cluster are written to the first computing node in a disorderly writing manner, and each message data is assigned a step operator identifier.

[0010] In an example of the above aspect, the data processing method further includes: in response to determining that the current iterative calculation ends normally or ends early, informing the scheduler through a corresponding interface to schedule the next round of iterative calculation.

[0011] According to another aspect of an embodiment of the present specification, a graph computing device based on asynchronous parallel processing is provided, the graph computing device is applied to a first computing node in a computing node cluster, the graph computing device includes: an iterative computing unit, in response to receiving message data required for a first step operator to perform a current iterative calculation from other computing nodes in the computing node cluster, using the received message data to perform the current iterative calculation of the first step operator, the first step operator being scheduled by a scheduler from a step operator set to the first computing node and one or more second computing nodes for parallel execution, the step operator set being obtained by expanding an executable grouped loop for iterative calculation in a graph logic execution plan; and a message data sending unit, asynchronously sending the current iterative calculation result of the first step operator as message data to the computing node to which the downstream step operator of the first step operator is scheduled, without waiting for all the second computing nodes to complete the current iterative calculation of the first step operator.

[0012] According to another aspect of an embodiment of the present specification, a graph computing engine is provided, comprising: a computing node cluster, each computing node in the computing node cluster comprising a graph computing device as described above; and a scheduler, which schedules step operators from a step operator set to a first computing node and one or more second computing nodes in the computing node cluster for parallel execution, wherein the step operator set is obtained by loop-expanding executable groups for iterative computing in a graph logic execution plan.

[0013] According to another aspect of an embodiment of the present specification, a graph computing device based on asynchronous parallel processing is provided, comprising: at least one processor; a memory coupled to the at least one processor; and a computer program stored in the memory, wherein the at least one processor executes the computer program to implement the graph computing method based on asynchronous parallel processing as described above.

[0014] According to another aspect of an embodiment of the present specification, a computer-readable storage medium is provided, which stores executable instructions, and when the instructions are executed, a processor executes the graph computing method based on asynchronous parallel processing as described above.

[0015] According to another aspect of an embodiment of the present specification, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to implement the graph computing method based on asynchronous parallel processing as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] A further understanding of the nature and advantages of the content of this specification may be achieved by referring to the following drawings. In the drawings, similar components or features may have the same reference numerals.

[0017] Figure 1 An example architecture diagram of a graph computing engine is shown.

[0018] Figure 2 An example schematic diagram showing a graph logic execution plan.

[0019] Figure 3 An example schematic diagram showing loop unrolling of a graph traversal operator.

[0020] Figure 4 An example schematic diagram of the traversal iterative calculation process based on the BSP model is shown.

[0021] Figure 5 An example flowchart of a graph computing method based on asynchronous parallel processing according to an embodiment of the present specification is shown.

[0022] Figure 6 An example schematic diagram of the traversal iterative calculation process based on the ASP model is shown.

[0023] Figure 7 An example schematic diagram of a shuffle data out-of-order writing process according to an embodiment of the present specification is shown.

[0024] Figure 8 An example block diagram of a graph computing device based on asynchronous parallel processing according to an embodiment of the present specification is shown.

[0025] Fig. 9 An example schematic diagram of a graph computing device implemented based on a computer system according to an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0026] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The functions and arrangements of the elements discussed can be changed without departing from the scope of protection of the contents of this specification. Various examples can omit, replace or add various processes or components as needed. For example, the described method can be performed in an order different from the described order, and various steps can be added, omitted or combined. In addition, the features described relative to some examples can also be combined in other examples.

[0027] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0028] The flowcharts used in this specification illustrate the operations implemented by the system according to some embodiments in this specification. It should be clearly understood that the operations of the flowcharts may not be implemented in sequence. On the contrary, the operations may be implemented in reverse order or simultaneously. In addition, one or more other operations may be added to the flowchart. One or more operations may be removed from the flowchart.

[0029] Graph computing engines usually have a distributed graph computing architecture. Figure 1 FIG. 1 shows an example architecture diagram of a graph computing engine 100. Figure 1 As shown, the graph computing engine 100 includes a scheduler 110 and a computing node cluster 120 . The computing node cluster 120 , for example, may include k computing nodes (workers) 120 - 1 to 120 - k .

[0030] After receiving the graph computing task, the scheduler 110 divides the graph computing task into multiple sub-computing tasks. For example, after receiving the graph computing task, the graph computing task can be converted into a graph logic execution plan through the underlying API provided by the graph computing engine. The graph logic execution plan can be, for example, a DAG (directed acyclic graph) composed of a series of graph computing operators with data dependencies. Figure 2 An example schematic diagram of a graph logic execution plan is shown. Figure 2 In the example, the graph logic execution plan includes operators source, map, traversalOP, flatmap, and sink with data dependencies.

[0031] The logical execution plan can be converted into a physical execution plan. In a physical execution plan, a set of repeatedly executable vertices (vertex) can be aggregated together to construct an executable group (ExecutionGroup). For example, in graph iteration calculations, graph iterations and their related graph iteration aggregation operators can be used as independent ExecutionGroups. In this way, the physical execution plan can be divided into one or more ExecutionGroups based on different execution modes and strategies. Each ExecutionGroup can be constructed by one or more vertices. Each executable group will eventually be converted into a basic unit Cycle that can be independently scheduled by the scheduler.

[0032] like Figure 2 As shown, the graph logic execution plan is divided into three cycles, namely, Cycle 1 consisting of operators source and map, Cycle 2 consisting of graph traversal operator traversalOP, and Cycle 3 consisting of operators flatmap and sink, where Cycle 2 is a Cycle involving iterative calculations. For Cycle 2 involving iterative calculations, its loop can be expanded into multiple step operator sets, each step operator set can be composed of one or more graph calculation operators, and each step operator set has a number.

[0033] Figure 3 An example schematic diagram of loop unrolling of a graph traversal operator is shown. Figure 3 In the example, the graph traversal operator traversalOP is loop expanded into 10 operators, of which operators 1:start and 10:end are automatically added start and end operators, and the remaining operators are actual step operators after loop expansion, which need to be distributed to computing nodes for iterative computing. Figure 3 As shown in the figure, the number of iterative calculation rounds after the loop is expanded includes 5 rounds of iterative calculations. The calculation performed by the step operator in the same dotted box is called one round of iterative calculations. The step operator in each round of iterative calculations can be scheduled by the scheduler to multiple workers for parallel execution. Each worker executes the calculation process of different step operators, or uses different data to execute the calculation process of the same step operator in parallel. One or more divided cycles or one or more expanded step operators can form a sub-computing task.

[0034] Subsequently, the scheduler dispatches the sub-computing tasks to each worker for parallel computing, thereby improving the efficiency of graph computing.

[0035] In the field of graph computing, the graph computing engine used is usually a graph computing engine based on the Bulk Synchronous Parallel (BSP) model. Figure 4 An example schematic diagram of the traversal iterative calculation process based on the BSP model is shown.

[0036] like Figure 4 As shown in the figure, in the BSP model, when performing iterative calculations, for multiple workers that perform the same iterative calculation, each worker will shuffle and distribute the message data obtained in this round of iterative calculations only after all workers have completed the round of iterative calculations. According to this processing method, each round of iterative calculations needs to wait for the barrier to be aligned, so that the iterative calculation time depends on the slowest worker, which limits the parallel processing capability of the BSP model.

[0037] In view of this, a graph computing scheme based on ASP is proposed. In this graph computing scheme, when performing iterative calculations, after the computing nodes participating in the current iterative calculation generate message data, they are immediately sent to the computing nodes participating in the next round of iterative calculations for rapid processing, without waiting for other computing nodes participating in the current iterative calculations to complete the iterative calculations. This eliminates the barrier dependency between iterations that occurs when performing iterative calculations based on the BSP model, reduces message backlogs, and implements traversal iterations in a pipelined manner.

[0038] The following will describe an ASP-based graph computing method, a graph computing device, and a graph computing engine according to an embodiment of the present specification with reference to the accompanying drawings.

[0039] The iterative calculations performed during graph calculations can usually include a specified number of rounds of iterative calculations, such as 100 rounds of iterative calculations. When performing graph calculations, the scheduler can batch schedule the specified number of rounds of iterative calculations to the computing nodes in a pipeline manner, such as 10 rounds of iterations. In each round of iterative calculations, the computing node where the step operator is located can determine whether the current round of iterative calculations is normal or ends early. For example, it can determine whether the current round of iterative calculations is normal or ends early by whether the step operator distributes message data to the downstream step operator. If it is determined that the current round of iterative calculations is normal or ends early, the scheduler is informed through the corresponding interface. Then, the scheduler will reschedule the next round of iterative calculations.

[0040] After the step operator used for iterative calculation is scheduled to the computing node, the computing node performs iterative calculation processing in the ASP manner.

[0041] Figure 5 FIG. 5 shows an example flow chart of a graph computing method 500 based on asynchronous parallel processing according to an embodiment of the present specification. Figure 5 In the example shown, the first computing node performs an iterative computation process of a first step operator that is scheduled to be executed at the first computing node. The first step operator is scheduled by the scheduler from a step operator set to the first computing node and other computing nodes (second computing nodes) in one or more computing clusters for parallel execution. The step operator set is obtained by loop expansion of executable groupings for iterative computation in the graph logic execution plan.

[0042] like Figure 5 As shown, at 510, it is determined whether the message data required for the first step operator to perform the current iterative calculation is received from other computing nodes in the computing node cluster. Other computing nodes refer to computing nodes scheduled by upstream operators. Upstream operators can be operators located in the previous cycle, for example, Figure 3 The Out operator numbered 2 in Cycle 2 has the upstream operator Figure 2 The map operator in Cycle 1 can also be the step operator in the same Cycle that is used to perform the previous round of iterative calculations. For example, Figure 3 The step operator union numbered 9 has upstream operators numbered 5 and 8. The message data required for the current iterative calculation may refer to the algorithm processing process of the first step operator that can be completed using only the received message data. For example, the current iterative calculation of the first step operator needs to execute a first calculation process based on data A and B and a second calculation process based on data C and D. In this case, if data A and B are received, even if data C and D are not received, it is considered that the message data required for the first step operator to execute the current iterative calculation is received. After receiving data A and B, the iterative calculation of the first step operator can be executed, thereby obtaining a partial iterative calculation result.

[0043] At 520 , after receiving the message data required by the first step operator to perform the current iterative calculation, the current iterative calculation of the first step operator is performed using the received message data.

[0044] At 530, after the current iterative calculation of the first step operator is completed, the current iterative calculation result of the first step operator is asynchronously sent as message data to the computing node to which the downstream step operator of the first step operator is scheduled, without considering whether the current iterative calculation of the first step operator is completed at other computing nodes that execute the first step operator in parallel, that is, there is no need to wait for the current iterative calculation of the first step operator to be completed at the second computing nodes that execute the first step operator in parallel.

[0045] Figure 6 FIG. 4 shows an example schematic diagram of the traversal iterative calculation process based on the ASP model. Figure 6As shown, the iterative calculations iteration1, iteration2, and iteration3 are scheduled to the computing nodes worker 1 to worker n in parallel, and iteration2 is the downstream iterative calculation of iteration1. After the iterative calculation of iteration1 is completed on worker 1, the iterative calculation result of iteration1 is immediately sent to worker 2 to worker n as message data, so that worker 2 to worker n can perform the iterative calculation of iteration 2 without waiting for worker 2 to worker n to complete the iterative calculation of iteration1. In addition, after worker 1 receives the message data required for the iterative calculation of iteration2 from other workers, it can immediately perform the iterative calculation of iteration2 at worker 1 without waiting for other workers to complete the iterative calculation and align the message data.

[0046] In some embodiments, the current iterative calculation result of the first step operator can be asynchronously sent to the computing node to which the downstream step operator of the first step operator is scheduled in the form of shuffle data.

[0047] In some embodiments, during data exchange, message data within the same executable group may be exchanged in a pipeline mode, and message data between different executable groups may be exchanged in a batch mode.

[0048] In some embodiments, when the current iterative calculation result of the first step operator is asynchronously sent to the computing node to which the downstream step operator of the first step operator is scheduled, the current iterative calculation result of the first step operator can be asynchronously sent to the computing node to which the downstream step operator of the first step operator is scheduled in a message window of a specified size. In other words, as long as the size of the iterative calculation result obtained at the first computing node reaches the specified window size, the iterative calculation result is sent as message data to the computing node to which the downstream step operator of the first step operator is scheduled. According to this message distribution method, message expansion can be avoided at the computing node to which the downstream step operator is scheduled.

[0049] In some embodiments, when performing data shuffle in a window size data distribution manner, an out-of-order write method may be supported, that is, message data of other computing nodes in the computing node cluster may be written to the first computing node in an out-of-order write method. Figure 7 An example schematic diagram of a shuffle data out-of-order writing process according to an embodiment of the present specification is shown.

[0050] exist Figure 7In the example, m in the identifier "dm-n" indicates the mth round of iterative data, and n indicates the nth window. For example, "d1-1" indicates the first round of iterative data sent using the first window. Figure 7 As shown in the figure, when the shuffer data is written out of order, the data written in different windows are cross-mixed and written, but the order of the final barriers remains consistent with the current order, that is, each barrier is immediately after the last data in the current round of iterative data.

[0051] When shuffle data is written out of order, each message data written is assigned a step operator ID to identify the step operator for subsequent processing. For the first batch of iterative data, the assigned step operator ID is empty or not assigned. For subsequent batches of iterative data, a specific step operator ID is assigned. When receiving shuffle data written out of order, the message will be split internally. Figure 3 As shown in the figure, for the first batch of iterative data, since they do not have a specific step id, they are directly handed over to node 1: start for processing. The message data processed by node 1: start will be assigned the step operator identifier "2: out". Similarly, the message data processed by node 2: out will be assigned the step operator identifier "3: out or 6: out". And so on. For subsequent batches of iterative data, since they all have step operator identifiers, the iterative data will be directly handed over to the step operator with the step operator identifier for processing.

[0052] In some embodiments, during each round of iteration, the computing node where the step operator is located can also determine whether the current iterative calculation is normal or ends early. For example, whether the step operator distributes message data to the downstream step operator can be used to determine whether the current iterative calculation is normal or ends early. If it is determined that the current iterative calculation is normal or ends early, the scheduler is notified through the corresponding interface to schedule the next round of iterative calculation.

[0053] Figure 8 FIG. 8 is a block diagram showing an example of a graph computing device 800 based on asynchronous parallel processing according to an embodiment of the present specification. Figure 8 As shown, the graph computing device 800 includes an iterative computing unit 810 and a message data sending unit 820.

[0054] The iterative calculation unit 810 is configured to respond to receiving message data required for the first step operator to perform the current iterative calculation from other computing nodes in the computing node cluster, and use the received message data to perform the current iterative calculation of the first step operator. The first step operator is scheduled by the scheduler from a step operator set to the first computing node and one or more second computing nodes for parallel execution. The step operator set is obtained by the scheduler by expanding the executable group loop for iterative calculation in the graph logic execution plan.

[0055] The message data sending unit 820 is configured to asynchronously send the current iterative calculation result of the first step operator as message data to the computing node scheduled to the downstream step operator of the first step operator, without waiting for the second computing node to complete the current iterative calculation of the first step operator.

[0056] In addition, the embodiments of this specification also provide a graph computing engine based on asynchronous parallel processing. The graph computing engine based on asynchronous parallel processing can be implemented as Figure 1 In some embodiments, the graph computing engine based on asynchronous parallel processing may include a computing node cluster and a scheduler. Each computing node in the computing node cluster includes: Figure 8 The graph computing device has a scheduler configured to expand the executable group loop for iterative computing in the graph logic execution plan into step operator computing, and schedule the step operators from the step operator set to the first computing node and one or more second computing nodes in the computing node cluster for parallel execution.

[0057] As above Figures 1 to 8 , describes a graph computing method based on asynchronous parallel processing, a graph computing device and a graph computing engine according to an embodiment of this specification. The above-mentioned graph computing device can be implemented by hardware, or by software, or by a combination of hardware and software.

[0058] Fig. 9 FIG. 9 is a schematic diagram showing an example of a graph computing device 900 implemented based on a computer system according to an embodiment of the present specification. Fig. 9 As shown, the graph computing device 900 may include at least one processor 910, a memory (e.g., a non-volatile memory) 920, a memory 930, and a communication interface 940, and the at least one processor 910, the memory 920, the memory 930, and the communication interface 940 are connected together via a bus 960. At least one processor 910 executes at least one computer-readable instruction stored or encoded in the memory (i.e., the above-mentioned element implemented in the form of software).

[0059] In one embodiment, computer executable instructions are stored in a memory, which when executed cause at least one processor 910 to: in response to receiving message data required for a first step operator to perform a current iterative calculation from other computing nodes in a computing node cluster, use the received message data to perform the current iterative calculation of the first step operator, the first step operator being scheduled by a scheduler from a step operator set to a first computing node and one or more second computing nodes for parallel execution, the step operator set being obtained by expanding an executable grouped loop for iterative calculation in a graph logic execution plan; and asynchronously sending the current iterative calculation result of the first step operator as message data to the computing node to which the downstream step operator of the first step operator is scheduled, without waiting for the second computing nodes to complete the current iterative calculation of the first step operator.

[0060] It should be understood that the computer executable instructions stored in the memory, when executed, cause at least one processor 910 to perform the above combined operations in various embodiments of the present specification. Figure 1-Figure 8 Describes the various operations and functions.

[0061] According to one embodiment, a program product such as a machine-readable medium (e.g., a non-transitory machine-readable medium) is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in the form of software), which, when executed by a machine, causes the machine to perform the above-mentioned combination of various embodiments of this specification. Figure 1-Figure 8 Specifically, a system or device equipped with a readable storage medium may be provided, on which a software program code implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system or device reads and executes the instructions stored in the readable storage medium.

[0062] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above-mentioned embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute a part of the present invention.

[0063] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0064] According to one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the processor performs the above combination of the various embodiments of this specification. Figure 1-Figure 8 Describes the various operations and functions.

[0065] Those skilled in the art should understand that the various embodiments disclosed above can be modified and altered in various ways without departing from the essence of the invention. Therefore, the protection scope of the present invention should be defined by the appended claims.

[0066] It should be noted that not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0067] In the above embodiments, the hardware unit or module can be realized by mechanical or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuit or logic (such as special processor, FPGA or ASIC) to complete the corresponding operation. The hardware unit or processor can also include programmable logic or circuit (such as general-purpose processor or other programmable processor), which can be temporarily set by software to complete the corresponding operation. Specific implementation (mechanical method or dedicated permanent circuit or temporary circuit) can be determined based on cost and time consideration.

[0068] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.

[0069] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A graph computing method based on asynchronous parallel processing, the graph computing method being executed by a first computing node in a computing node cluster, the graph computing method comprising: In response to receiving message data required for a first step operator to perform a current iterative calculation from other computing nodes in the computing node cluster, using the received message data to perform the current iterative calculation of the first step operator, the first step operator being scheduled by a scheduler from a step operator set to the first computing node and one or more second computing nodes for parallel execution, the step operator set being obtained by loop expansion of executable groups for iterative calculation in a graph logic execution plan; as well as The current iterative calculation result of the first step operator is asynchronously sent as message data to the computing node to which the downstream step operator of the first step operator is scheduled, without waiting for the second computing node to complete the current iterative calculation of the first step operator.

2. The graph computing method according to claim 1, wherein: The current iterative calculation result of the first step operator is asynchronously sent in the form of shuffle data to the computing node to which the downstream step operator of the first step operator is scheduled.

3. The graph computing method according to claim 1, wherein: Asynchronously sending the current iterative calculation result of the first step operator to the computing node to which the downstream step operator of the first step operator is scheduled includes: The current iterative calculation result of the first step operator is asynchronously sent to the computing node to which the downstream step operator of the first step operator is scheduled in a message window of a specified size.

4. The graph computing method according to claim 1, wherein: Message data within the same executable group is exchanged in a pipeline mode, and message data between different executable groups is exchanged in a batch mode.

5. The graph computing method according to claim 3, wherein: The message data of other computing nodes in the computing node cluster are written into the first computing node in a disorderly writing manner, and each message data is assigned a step operator identifier.

6. The graph computing method according to claim 1, further comprising: In response to determining that the current iterative calculation ends normally or ends early, the scheduler is informed through a corresponding interface to schedule the next round of iterative calculation.

7. A graph computing device based on asynchronous parallel processing, the graph computing device being applied to a first computing node in a computing node cluster, the graph computing device comprising: an iterative computing unit, in response to receiving message data required for a first step operator to perform a current iterative computing from other computing nodes in the computing node cluster, using the received message data to perform the current iterative computing of the first step operator, the first step operator being scheduled by a scheduler from a step operator set to the first computing node and one or more second computing nodes for parallel execution, the step operator set being obtained by loop expansion of executable groups for iterative computing in a graph logic execution plan; as well as The message data sending unit asynchronously sends the current iterative calculation result of the first step operator as message data to the computing node scheduled to the downstream step operator of the first step operator, without waiting for the second computing node to complete the current iterative calculation of the first step operator.

8. A graph computing engine, comprising: A computing node cluster, each computing node in the computing node cluster comprising the graph computing device as claimed in claim 7; as well as A scheduler schedules step operators from a step operator set to a first computing node and one or more second computing nodes in a computing node cluster for parallel execution, wherein the step operator set is obtained by loop-expanding executable groups for iterative calculations in a graph logic execution plan.

9. A graph computing device based on asynchronous parallel processing, comprising: at least one processor; a memory coupled to the at least one processor; as well as The computer program stored in the memory, and the at least one processor executes the computer program to implement the graph computing method based on asynchronous parallel processing as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium storing executable instructions, which, when executed, enable a processor to perform the graph computing method based on asynchronous parallel processing as described in any one of claims 1 to 6.

11. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the graph computing method based on asynchronous parallel processing as claimed in any one of claims 1 to 6.

Images