Data processing method based on dataflow graph, computing apparatus and computing device cluster

Abstract

The present disclosure provides a data processing method based on a dataflow graph, a computing apparatus and a computing device cluster, which may be used in the field of computer technology and solve the technical defect that the existing dataflow graph is immutable in the program execution stage and therefore cannot be used for object-oriented program. The method includes: obtaining a sub dataflow graph, where the sub dataflow graph is a copy of a portion of the dataflow graph, and the sub dataflow graph includes one or more first nodes and one or more first edges, the one or more first nodes are used for data processing, and the one or more first edges are channels between the one or more first nodes; and processing data based on the sub dataflow graph.

Description

DATA PROCESSING METHOD BASED ON DATAFLOW GRAPH, COMPUTING APPARATUS AND COMPUTING DEVICE CLUSTERTECHNICAL FIELD

[0001] The present disclosure relates to the field of computer technology, and in particular, to a data processing method based on a dataflow graph, a computing apparatus, a computing device cluster and a computer-readable storage medium.BACKGROUND

[0002] In computer systems, the dataflow architecture is different from the traditional von Neumann (control flow) architecture. The control flow architecture executes instructions sequentially, while the dataflow architecture is driven by data and doesn’t have a program counter. A program may be considered as a dataflow graph (DFG), where nodes are "operators", and directed edges between them are "channels" through which immutable data values travel. An operator is evaluated when data is present on all input channels. The completion of evaluation makes the resulting data available to the operators at the end of output channels. It is noted that the program, represented as a DFG, is an immutable memory artifact, and thus data being transferred between operators of the DFG don’t change.

[0003] However, in object-oriented program, the objects may not be transferred through the edges of a DFG, because they are parts of the DFG. On the other hand, the DFG may not accept any changes to itself in runtime, due to its immutability.SUMMARY

[0004] This present disclosure provides a data processing method based on a dataflow graph, a computing apparatus, a computing device cluster and a computer-readable storage medium, to realize the mutability of DFG, thereby extending the classical dataflow architecture, and realizing the transmission of objects in an object-oriented program.

[0005] According to a first aspect, a data processing method based on a dataflow graph is described. The method includes: obtaining a sub dataflow graph, where the sub dataflow graph is a copy of a portion of the dataflow graph, and the sub dataflow graph includes one or more first nodes and one or more first edges, the one or more first nodes are used for data processing, and the one or more first edges are channels between the one or more first nodes; and processing data based on the sub dataflow graph.

[0006] Based on the method provided, in the process of processing data based on the dataflow graph, a processor may create a copy of a portion of the dataflow graph and process data based on that copy. For example, a sub dataflow graph which is a copy of a portion of the dataflow graph may be created, and the data may be processed based on the sub dataflow graph. The method may not only change the dataflow graph, but also utilize the mutable dataflow graph to process the data, thus solving the technical defect that the existing dataflow graph is immutable in the program execution stage and therefore cannot be used for object-oriented program.

[0007] In some embodiments, the method further includes: creating one or more second edges, where the one or more second edges are channels between one or more second nodes in the dataflow graph and at least one of the one or more first nodes, and the processing data is further based on the one or more second edges and the one or more second nodes.

[0008] In this method, the one or more second edges are used to connect one or more second nodes in the dataflow graph and at least one of the one or more first nodes, so that the data and / or objects may be passed on smoothly.

[0009] In some embodiments, the portion of the dataflow graph indicates transformations defined by a predefined class in the dataflow graph.

[0010] Although the portion of the dataflow graph has a mutable structure, it may exist in the dataflow graph steadily, which may describe transformations defined by the predefined class in the dataflow graph.

[0011] In some embodiments, the one or more first nodes are copies of one or more nodes in the portion of the dataflow graph, and the one or more first edges are copies of one or more edges in the portion of the dataflow graph.

[0012] In this scenario, the sub dataflow uses the portion of the dataflow graph as a template, and the first nodes and the first edges may be obtained. The first nodes and the first edges are mutable, thus ensuring the mutability of the sub dataflow graph.

[0013] In some embodiments, obtaining the sub dataflow graph includes: obtaining the sub dataflow graph when the portion of the dataflow graph is invoked.

[0014] For example, when the portion of the dataflow graph is invoked, which means a processor may begin to utilize the sub dataflow graph for data processing, the invocation of the portion of the dataflow graph may be regarded as a signal to obtain the sub dataflow graph.

[0015] In some embodiments, the data processing method based on the dataflow graph further includes: destroying the sub dataflow graph and the one or more second edges.

[0016] After the calculation is completed, a "garbage collection" may be used to destroy the sub dataflow graph and the one or more second edges, and this operation may reduce system cache for the next computation.

[0017] In some embodiments, destroying the sub dataflow graph and the one or more second edges includes: destroying the one or more first nodes, the one or more first edges and the one or more second edges.

[0018] In the process of destroying the sub dataflow graph and the one or more second edges, one or more first nodes and one or more first edges connected to the first nodes and one or more second edges may be destroyed, which may reduce the waste of memory resources.

[0019] In some embodiments, destroying the one or more first nodes, the one or more first edges and the one or more second edges includes: destroying the one or more first edges and the one or more second edges automatically when destroying the one or more first nodes.

[0020] This mode means that an implicit garbage collection algorithm may to be used, and the algorithm determines which nodes are no longer needed, and then a processor may automatically asynchronously delete these nodes in the sub dataflow graph according to the algorithm.

[0021] In some embodiments, destroying the one or more first nodes, the one or more first edges and the one or more second edges includes: destroying the one or more first nodes in response to one or more node destruction instructions.

[0022] In this scenario, the garbage collection mode may be explicit garbage collection. This method is flexible and may use operators to delete unwanted nodes autonomously.

[0023] According to a second aspect, a computing apparatus is described, which includes: an obtaining unit configured to obtain a sub dataflow graph, where the sub dataflow graph is a copy of a portion of the dataflow graph, and the sub dataflow graph includes one or more first nodes and one or more first edges, the one or more first nodes are used for data processing, and the one or more first edges are channels between the one or more first nodes; and process data based on the sub dataflow graph.

[0024] In some embodiments, the processing unit is configured to: create one or more second edges, where the one or more second edges are channels between one or more second nodes inthe dataflow graph and at least one of the one or more first nodes; and process data based on the one or more second edges, the one or more second nodes and the sub dataflow graph.

[0025] In some embodiments, the portion of the dataflow graph indicates transformations defined by a predefined class in the dataflow graph.

[0026] In some embodiments, the one or more first nodes are copies of one or more nodes in the portion of the dataflow graph, and the one or more first edges are copies of one or more edges in the portion of the dataflow graph.

[0027] In some embodiments, the obtaining unit is configured to obtain the sub dataflow graph when the portion of the dataflow graph is invoked.

[0028] In some embodiments, the computing apparatus further includes a destroying unit configured to destroy the sub dataflow graph and the one or more second edges.

[0029] In some embodiments, the destroying unit is configured to destroy the one or more first nodes, the one or more first edges and the one or more second edges.

[0030] In some embodiments, the destroying unit is configured to destroy the one or more first edges and the one or more second edges automatically when destroying the one or more first nodes.

[0031] In some embodiments, the destroying unit is configured to destroy the one or more first nodes in response to one or more node destruction instructions.

[0032] According to a third aspect, a computing device cluster is described. The computing device cluster includes at least one computing device, where the computing device includes a processor and a memory coupled with the processor, where the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device cluster executes the method in the first aspect or any possible design of the first aspect.

[0033] According to a fourth aspect, a computer-readable storage medium is described. The computer-readable storage medium has instructions stored thereon which, when executed by a computing device, cause the computing device to perform the above-mentioned data processing method based on the dataflow graph.

[0034] According to a fifth aspect, a computer program is described which, when executed by a computing device, causes the computing device to perform.

[0035] According to a sixth aspect, a computer program product is described. The computer program product includes computer instructions, when the computer instructions are executed by a computing device, the above-mentioned data processing method based on the dataflow graph is implemented.BRIEF DESCRIPTION OF THE DRAWINGS

[0036] For a better understanding of the various described embodiments, reference should be made to the Detailed Description below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures.

[0037] FIG. 1 is an example dataflow program;

[0038] FIG. 2 is an architecture of an example dataflow machine;

[0039] FIG. 3 is a flow chart of a data processing method based on a dataflow graph, in accordance with some embodiments of the present disclosure;

[0040] FIGS. 4A to 4C show a structural diagram for a data processing method based on a dataflow graph, in accordance with some embodiments of the present disclosure;

[0041] FIGS. 5A to 7C show an example process of creating a sub dataflow graph, in accordance with some embodiments of the present disclosure;

[0042] FIG. 8 is an architecture of a dataflow machine, in accordance with some embodiments of the present disclosure;

[0043] FIG. 9 shows an example Java program that calculates an area of a circle;

[0044] FIGS. 10A to 10C show an example process of a data processing method, in accordance with some embodiments of the present application;

[0045] FIG. 11 is a block diagram of a computing apparatus based on a dataflow graph, in accordance with some embodiments of the present disclosure;

[0046] FIG. 12 is a schematic diagram of an architecture of a computing device, in accordance with some embodiments of the present disclosure;

[0047] FIG. 13 is a schematic diagram of an architecture of a computing device cluster, in accordance with some embodiments of the present disclosure; and

[0048] FIG. 14 is a schematic diagram of a connection between computing devices over a network, in accordance with some embodiments of the present disclosure.DETAILED DESCRIPTION

[0049] Technical solutions in embodiments of the present disclosure will be described clearly and completely with reference to the accompanying drawings below. Obviously, the described embodiments are merely some but not all embodiments of the present disclosure. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present disclosure shall be included in the protection scope of the present disclosure.

[0050] Unless the context requires otherwise, throughout the description and the claims, the term "comprise" and other forms thereof such as the third-person singular form "comprises" and the present participle form "comprising" are construed as open and inclusive meanings, i.e., "including, but not limited to". In the description, the terms such as "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" are intended to indicate that specific features, structures, materials or characteristics related to the embodiment(s) or example(s) are included in at least one embodiment or example of the present disclosure. Schematic representations of the above terms do not necessarily refer to the same embodiment(s) or examples(s). In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any suitable manner.

[0051] Hereinafter, the terms "first" and "second" are used for descriptive purposes only, and are not to be construed as indicating or implying the relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the terms "a / the plurality of' and "multiple" means two or more unless otherwise specified.

[0052] In the description of some embodiments, the terms "coupled" and "connected" and derivatives thereof may be used. For example, the term "connected" may be used in the description of some embodiments to indicate that two or more components are in direct physical or electrical contact with each other. For another example, the term "coupled" may be used in the description of some embodiments to indicate that two or more components are in direct physical or electrical contact. However, the term "coupled" may also mean that two or more components are not in direct contact with each other, but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the content herein.

[0053] The phrase "A and / or B" includes the following three combinations: only A, only B, and a combination of A and B.

[0054] Terms used in embodiments of the present disclosure are described below for ease of the understanding.

[0055] Dataflow graph (DFG) is a graph which represents data dependencies between a number of operations. A DFG maps out the flow of information for any process or system. It uses defined symbols like rectangles, circles and arrows, plus short text labels, to show data inputs, outputs, storage points and the routes between each destination. DFGs may range from simple, even hand-drawn process overviews, to in-depth, multi-level DFGs that dig progressively deeper into how the data is handled. They may be used to analyze an existing system or model a new one. Like all the best diagrams and charts, a DFG may often visually “say” things that would be hard to explain in words.

[0056] FIG. 1 shows an example dataflow program, which is a graph with operators as nodes and data channels as edges between the nodes. An operator is a node that represents a data transformation or computation having one or more input ports and one or more output ports; the data flows from the input ports to the output ports, and the operator performs some transformation on the data as it flows through. In FIG. 1, theses operators include an add operator representing an add operation, an operator div representing a division operation, and an operator mul representing multiply operation.

[0057] For example, the operator div has two input ports which receive input data Xi (i.e., 32) and X2 (i.e., 16), and one output port which outputs 2. The operator mul has two input ports which receive input data X2 (i.e., 16) and X3 (i.e., 11), and one output port which outputs 176. The operator add includes two input ports which receive data output by the operators div and add, and one output port which output 178.

[0058] The dataflow graph may be performed in a microarchitecture (also referred to as "microprocessor architecture") which is the way a given instruction set architecture is implemented in a particular processor.

[0059] FIG. 2 shows a dataflow architecture originally proposed by Dennis et al. in 1974. The dataflow architecture may also be referred to as a dataflow machine. The dataflow machine may give control to any particular operator when the data it required are already available in the input channels. Thus, the order of execution is non-deterministic.

[0060] In addition, in computer systems, the dataflow architecture is different from the traditional von Neumann (control flow) architecture. The control flow architecture executes instructions sequentially, while the dataflow architecture is driven by data and doesn’t have a program counter.

[0061] As shown in FIG. 2, the dataflow architecture includes an arbitration network, operators, a distribution network, operation units, etc.

[0062] In this architecture, an operation unit (OU) is a computational unit that performs a specific operation on the data in an instruction cell (IC), which is a small unit of memory that stores an instruction and its operands. The IC is responsible for controlling the execution of the instruction, and it also provides the operands to the functional units that perform the actual computation.

[0063] In this dataflow architecture, data token is a piece of data transferred by the dataflow machine from a DFG through arbitration network to operation units and then to the distribution network, containing meta information on top of the data itself for helping coordinate the processing and dispatching the data.

[0064] It is worth noting that in FIG. 2, the DFG created by the dataflow machine is an immutable memory artifact, and thus is not changing its structure during the entire lifecycle of a program. In this case, every operator is a micro-program that inputs data tokens and produces new data tokens, and the data tokens are immutable, which means that the data being transferred between operators don’t change. However, the entire pool of tokens is mutable: when new data are generated by operators, new tokens are taken from the pool, filled up with the data, and sent to the arbitration network, the operation units and then to the distribution network.

[0065] In some scenarios, an object-oriented program, which is a programming paradigm that treats data as an object that has both state and behavior, is mapped into a DFG and then it is executed on a dataflow machine: objects, each of which is a self-contained entity that has state and behavior (the state of an object is represented by its data attributes, and the behavior of an object is represented by its methods), as operators and method calls as channels between them, thus inheriting some parallelism from the object hierarchy itself.

[0066] However, objects, unlike data, may not be transferred through the edges of a DFG. The reason for this is that in the dataflow graph, an object is a combination of operators each of which is a method or field of the object. That is, the objects are parts of the DFG. However, datatransmited through the edge between nodes is a byte link, which has a simpler and flatter structure than the object. On the other hand, the DFG may not accept any changes to itself in runtime, due to its immutable nature.

[0067] To improve at least one of the problems described above, the embodiments of the present disclosure provide a data processing method based on a dataflow graph, and the data processing method is described in combination with examples below.

[0068] FIG. 3 is a flow chart of a data processing method based on a dataflow graph, in accordance with some embodiments of the present disclosure. The data processing method may be performed by a microarchitecture (also referred to as microprocessor architecture, dataflow machine), a processor, a microprocessor, a multiprocessor, or the like.

[0069] In some embodiments, as shown in FIG. 3, the data processing method based on the dataflow graph includes steps 310 to 320.

[0070] In step 310, a sub dataflow graph is obtained.

[0071] The sub dataflow graph is a copy of a portion of the dataflow graph, and the sub dataflow graph includes one or more first nodes and one or more first edges. The one or more first nodes are used for data processing, and the one or more first edges are channels between the one or more first nodes.

[0072] In step 320, data is processed based on the sub dataflow graph.

[0073] In the embodiments of the present disclosure, when input data or objects enter the dataflow graph, the sub dataflow graph which is a copy of the portion of the dataflow graph may be obtained. In this case, the dataflow graph includes the portion of the dataflow graph and the sub dataflow graph. That is, the dataflow graph is mutable. In the mutable dataflow graph, for object-oriented programming, an object may be represented as one or more first nodes as well as one or more first edges between the nodes, and the transmission of objects may be achieved through the transmission of identities (IDs) of the nodes.

[0074] In some embodiments, the method further includes: creating one or more second edges. The one or more second edges are channels between one or more second nodes in the dataflow graph and at least one of the one or more first nodes, and processing data is further based on the one or more second edges and the one or more second nodes.

[0075] In this method, the one or more second edges are used to connect one or more second nodes in the dataflow graph and at least one of the one or more first nodes, so that the data or objects may be passed on smoothly.

[0076] In some embodiments, the portion of the dataflow graph indicates transformations defined by a predefined class in the dataflow graph.

[0077] Although the portion of the dataflow graph is a mutable structure, it may exist in the dataflow graph steadily, which may describe transformations defined by the predefined class in the dataflow graph. The portion of the dataflow graph, defining the implement of the computational methods, may not participate in the computational process.

[0078] The predefined class includes one or more predefined computational methods, the implementation of which is defined by the sub dataflow graph. The computational methods include, but are not limited to: addition, subtraction, multiplication, division, multiplication, square, etc.

[0079] In some embodiments, the one or more first nodes are copies of one or more nodes in the portion of the dataflow graph, and the one or more first edges are copies of one or more edges in the portion of the dataflow graph.

[0080] The one or more first nodes are operators, and the one or more first edges are channels between the one or more first nodes. In the sub dataflow graph, the one or more first edges may be used as channels for data or objects to circulate. In addition, the sub dataflow graph not only defines the computational methods, but also utilizes the one or more first nodes and the one or more second edges to participate in the computational process.

[0081] In some embodiments, obtaining the sub dataflow graph includes: obtaining the sub dataflow graph when the portion of the dataflow graph is invoked.

[0082] For example, when an area of a circle needs to be calculated, the portion of the dataflow graph for the calculation of the area of the circle corresponding to the calculation of the circular are is called. In this case, an operator may send instructions to a processor. The processor may use the portion of the dataflow graph as a template, create nodes and edges of a sub dataflow graph, and use the sub dataflow graph for the calculation of the area of the circle.

[0083] For example, when operator A needs to pass objects to operator B, operator A may send instructions to a processor, and the processor creates multiple nodes and edges that represent theobjects to be delivered, and the processor sends the IDs of the nodes to operator B, and the nodes corresponding to the IDs is the objects transmitted.

[0084] The sub dataflow graph may be obtained in step 310 in number of ways, which are described below by way of examples.

[0085] FIGS. 4A to 4C show an example dataflow graph and corresponding hash mapping tables that are used to maintain information about the dataflow graph.

[0086] FIG. 4A shows an example dataflow graph. As shown in FIG. 4A, the dataflow graph includes three nodes vi, V2 and vs and two edges between the nodes.

[0087] FIG. 4B shows an example hash mapping table which is used to maintain information about the nodes in the dataflow graph. The left column shows identities (IDs) of the nodes, and the right column shows operators in binary format.

[0088] FIG. 4C shows an example hash mapping table which is used to maintain information about edges between nodes in the dataflow graph. The left column shows IDs of start nodes, the right column shows IDs of arriving nodes, and each row shows information corresponding to a directed edge.

[0089] FIGS. 5A to 7C show a process of creating a sub dataflow graph.

[0090] As shown in FIG. 5A, the dataflow graph includes four nodes vi to V4 and two edges, and the node V4 is a copy of node vi.

[0091] In some examples, the process of creating node V4 may be as follows. For a dataflow machine, which is a processor that evaluates the operators of the DFG, an operator may send an instruction to the processor or a supplementary processor for creating a copy of node vi. The processor or the supplementary processor may perform operations on the dataflow graph after receiving the instruction, so as to realize the addition of the node.

[0092] For example, the operator / sends an ADD instruction to the processor. After receiving the instruction, the processor creates a node by inserting a row into the hash mapping table in FIG. 4B. The updated hash mapping table is as shown in FIG. 5B, which shows the new fourth row reflecting the information of the new node V4. It is noted that as shown in FIG. 5C, there is no ID of an arriving node corresponding to the new node since there is no edge connected to the new node.

[0093] As shown in FIG. 6A, the dataflow graph includes five nodes vi to vs and three edges. The node V4 is a copy of node vi, the node vj is a copy of node V2, and the edge between the nodes V4 and vs is a copy of the edge between the nodes vi and V2.

[0094] As shown FIG. 6B, the process of creating the node vj may be referred to the process of creating the node V4, which will not be repeated here for the sake of simplicity. The process of creating an edge will be described below.

[0095] In some examples, the operator f may send a BIND instruction to the processor, which creates an edge between the nodes V4 and Vj after receiving the instruction by inserting a row into the corresponding hash mapping table in FIG. 4C. The updated hash mapping table is as shown in FIG. 6C, which shows a new row reflects a directed edge between the nodes v4 and v5.

[0096] As shown in FIG. 7A, the dataflow graph includes six nodes vi to vs and four edges. The node V4 is a copy of node vi, the node vs is a copy of node V2, and the node v<$ is a copy of node vs. The edge between the nodes v4 and vj is a copy of the edge between the nodes vi and V2, and the edge between the nodes vs and v^is a copy of the edge between the nodes V2 and vj.

[0097] As shown FIG. 7B, the process of creating the node vs may be referred to the process of creating the node V4, which will not be repeated here for the sake of simplicity. As shown FIG. 7C, the process of creating the edge between the nodes vs and V6 may be referred to the process of creating the edge between the nodes V4 and v , which will not be repeated here for the sake of simplicity. After the edge between the nodes vs and vs is created, a same sub dataflow graph is created.

[0098] In this case, as shown in FIG. 7A, the dataflow graph includes a portion of the dataflow graph and a sub dataflow graph. The portion of the dataflow graph includes three nodes vi to vs and edges between the nodes vi to vs. The sub dataflow graph includes three first nodes V4 to vs and first edges between the second nodes. And the operator / is a second node.

[0099] FIG. 8 shows an architecture of a dataflow machine, in accordance with some embodiments of the present disclosure.

[0100] As shown in FIG. 8, the original dataflow machine shown in FIG. 2 is extended by the processor, and operators N+l and N+2, which both have a function of the operator / described above, are added. Unlike the operators in the original dataflow machine, the new operators N+land N+2 may be used not only for data calculation, but also for creating new nodes and edges between operators or deleting the new added nodes and edges.

[0101] In some embodiments, the data processing method based on the dataflow graph includes: destroying the sub dataflow graph and the one or more second edges.

[0102] In some examples, in addition to support for creating nodes and edges, the operator in the dataflow machine also supports the deletion of the new added nodes and edges, which may also be referred to as "garbage collection".

[0103] In some implements, destroying the sub dataflow graph and the one or more second edges includes: destroying the one or more first nodes, the one or more first edges and the one or more second edges.

[0104] For example, destroying the sub dataflow graph includes: destroying any one of the one or more first nodes; destroying each of the one or more first nodes; or destroying some of the one or more first nodes.

[0105] For example, destroying the sub dataflow graph includes: destroying any one of the one or more first edges; destroying each of the one or more first edges; or destroying some of the one or more first edges.

[0106] For example, destroying the one or more second edges includes: destroying any one of the one or more second edges; destroying each of the one or more second edges; or destroying some of the one or more second edges.

[0107] In some embodiments, destroying the one or more first nodes, the one or more first edges and the one or more second edges includes: destroying the one or more first edges and the one or more second edges automatically when destroying the one or more first nodes.

[0108] For example, as shown in FIG. 7B, if the processor deletes the information about the nodes vj and V6 in the hash mapping table, then as shown in FIG. 7C, the information about the edge between the nodes vj and V6 in the hash mapping table may be deleted automatically without instructions.

[0109] In some embodiments, destroying the one or more first nodes, the one or more first edges and the one or more second edges includes: destroying the one or more first nodes in response to one or more node destruction instructions.

[0110] There are several ways to achieve destruction of nodes and edges in response to one or more node destruction instructions, two of which are illustrated below as examples. One way may be referred to as explicit deletion and the other may be referred to as implicit deletion.

[0111] For example, in the explicit deletion, a classic reference counting algorithm may be used. The implementation of the counting algorithm relies on a reference counter, which is a special type of counter. The processor may configure a reference counter for each node. When a node is used for calculation, a value of a reference counter of the node is increased by one; when a node is no longer used for calculation, a value of a reference counter of the node is reduced by one. When a value of a reference counter of a node is 0, it means that the node is no longer used. The operator f may send an instruction such as DELETE to the processor. The processor processes the instruction and deletes the node. When nodes are deleted, edges between the nodes are automatically deleted.

[0112] For another example, in the implicit deletion, a "garbage collection" algorithm is used, which is as follows.

[0113] In some examples, the garbage collection algorithm is the Mark-Sweep algorithm, and the algorithm is executed as follows. When the available memory space for storing nodes and edges is exhausted, the processor may do two jobs: the first is marking and the second is sweeping. During the marking phase, the processor may perform a traversal of all nodes and mark the nodes which may be used for calculation currently or in the future as reachable nodes. During the sweeping phase, the processor may perform a traversal of all nodes. If the processor finds that some nodes are not marked as reachable, the processor may delete the nodes and delete the edges between the nodes automatically.

[0114] FIG. 9 shows an example Java program that calculates an area of a circle. As shown in FIG. 9, a class called Circle is predefined, which contains a property r that represents the radius of the circle. A function, i.e., Circle(double r), is defined in the class to initialize the radius of the circle. There is also a method in the class Circle, i.e., area(), which is used to calculate the area of a circle. Then a method called f() is defined in the program, which creates a new Circle object and returns a circle with a radius of 42.0. At the end of the program, a Circle object is obtained by calling the f() method, and then the area() method of the object is called to calculate its area, assigning the result to variable a.

[0115] It is noticed that for object-oriented program, when the DFG is immutable, objects may not be transmitted through the edge of the DFG. However, in the application scenario described in the embodiments of the present disclosure, objects may be transmitted in the DFG to realize the calculation of the area of the circle.

[0116] FIGS. 10A to 10C shows an example working process of a mutable dataflow graph when it is used to run the program of calculating the area of the circle shown in FIG. 9.

[0117] FIG. 10A shows a dataflow graph for the calculation of the area of the circle. As shown in FIG. 10A, a portion of the dataflow graph, which includes three nodes (i.e., ctor, r and area), and two edges between the three nodes.

[0118] As shown in FIG. 10A, the dataflow graph includes a portion of the dataflow graph of a predefined class, which is used for describing the calculation of an area of a circle, and the calculation method is represented by three nodes of the portion of the dataflow graph, including an operator ctor, a circle radius operator r and a circle area operator area. In addition, the portion of the dataflow graph further includes two edges between the three nodes.

[0119] In addition, in FIG. 10A, three other operators are also shown, which are operator main, operator f and operator a. The operator main represents a starting node; the operator main points to the operator f. Operator f is used to create new nodes and edges, and the operator a stores output data. These two operators are second nodes. In this case, the operator a is not connected to other nodes, indicating that the portion of the dataflow graph is not directly involved in the actual computational process.

[0120] FIG. 10B shows another dataflow graph obtained after the operator f creates an instance on the basis of a predefined class.

[0121] In FIG. 10B, the operator f may use the portion of the dataflow graph as a template to create a sub dataflow graph. Operator f may instantiate predefined classes and send an instruction to the processor, which receives the instruction and creates three nodes. Then the processor connects the created nodes with directed edges. In this case, the dataflow graph includes the portion of the dataflow graph and a sub dataflow graph. The portion of the dataflow graph includes three nodes (i.e., ctor, r and area) and two edges on the left, and the sub dataflow graph includes three first nodes (i.e., ctor, r and area) and two first edges between the nodes on the right. The dataflow graph further includes two second nodes (i.e., / and a) and edges betweennodes in the sub dataflow graph and the two second nodes (i.e., an edge between operators ctor and f and an edge between operators area and a).

[0122] As shown in FIG. 10B, when the predefined class is instantiated, the operator ctor is automatically called by the processor. The operator f passes 42 as input to the operator ctor. The operator ctor goes on to pass 42 to the operator r, which passes 42 as the radius of the circle to the operator area. The operator area uses the radius of the circle to calculate the area of the circle as 5538, and passes 5538 as the output to operator a.

[0123] In some embodiments, after the calculation is completed, the instance of the predefined class is destroyed, and the three nodes and the directed edge created by the operator are deleted, as shown in FIG. 10C, which shows a dataflow graph after the program execution is completed.

[0124] Some embodiments provide a computing device for a data processing method based on a dataflow graph. It should be noted that the device embodiments are the same or similar to the method embodiments, and the relevant content may be referred to the description of the method embodiments.

[0125] Some embodiments provide a computing apparatus, as shown in FIG. 11, which may include: an obtaining unit 1101 configured to obtain a sub dataflow graph, where the sub dataflow graph is a copy of a portion of the dataflow graph, and the sub dataflow graph includes one or more first nodes and one or more first edges, the one or more first nodes are used for data processing, and the one or more first edges are channels between the one or more first nodes; and a processing unit 1102 configured to process data based on the sub dataflow graph.

[0126] In some embodiments, the processing unit 1102 is configured to: create one or more second edges, where the one or more second edges are channels between one or more second nodes in the dataflow graph and at least one of the one or more first nodes; and process data based on the one or more second edges, the one or more second nodes and the sub dataflow graph.

[0127] In some embodiments, the portion of the dataflow graph indicates transformations defined by a predefined class in the dataflow graph.

[0128] In some embodiments, the one or more first nodes are copies of one or more nodes in the portion of the dataflow graph, and the one or more first edges are copies of one or more edges in the portion of the dataflow graph.

[0129] In some embodiments, the obtaining unit 1101 is configured to obtain the sub dataflow graph when the portion of the dataflow graph is invoked.

[0130] In some embodiments, as shown in FIG. 11, the computing apparatus further includes a destroying unit 1103 configured to destroy the sub dataflow graph and the one or more second edges.

[0131] In some embodiments, the destroying unit 1103 is configured to destroy the one or more first nodes, the one or more first edges and the one or more second edges.

[0132] In some embodiments, the destroying unit 1103 is configured to destroy the one or more first edges and the one or more second edges automatically when destroying the one or more first nodes.

[0133] In some embodiments, the destroying unit 1103 is configured to destroy the one or more first nodes in response to one or more node destruction instructions.

[0134] A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments, units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on particular applications and design constraints of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application.

[0135] It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiment. Details are not described herein again.

[0136] In several embodiments, it will be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example.

[0137] The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0138] In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

[0139] FIG. 12 is a schematic diagram of an architecture of a computing device according to an embodiment of the present application. The computing device 1200 may be a server or a computer or other device with computing capabilities. The computing device 1200 shown in FIG. 12 includes: a processor 1220, a memory 1230, a transceiver 1240, and a bus 1210. The processor 1220, the memory 1230, and the transceiver 1240 are in communication with each other via the bus 1210. The computing device 1200 may be a server or a terminal device. The present application does not limit the number of processors, and memories in the computing device 1200.

[0140] The bus 1210 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Buses may be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in FIG. 12, but it does not mean that there is only one bus or one type of bus. The bus 1210 may include pathways for transferring information between various components of the computing device 1200 (e.g., the memory 1230, the processor 1220, or the transceiver 1240).

[0141] The processor 1220 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0142] The memory 1230 may include volatile memory, such as random access memory (RAM). The processor 1220 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a mechanical hard disk (HDD), or a solid state drive (SSD).

[0143] An executable program code is stored in the memory 1230, and the processor 1220 executes the executable program code to implement the functions of the aforementioned obtaining unit 1101, the processing unit 1102, or the destroying unit 1103, so as to implement the data processing method described in the above embodiment. That is, the memory 1230 has instructions on it for executing the data processing method described in the above embodiment.

[0144] The structure of the computing device 1200 enumerated above is only an exemplary illustration, and the present application is not limited thereto. The computing device 1200 of embodiments of the present application includes a variety of hardware found in computer systems in the prior art, e.g., the computing device 1200 also includes memory other than thememory 1230, such as, for example, a disk memory, and the like. It should be understood by those skilled in the art that the computing device 1200 may also include other devices necessary to achieve proper operation. In addition, depending on the specific needs, one of skill in the art should understand that the above-described computing device 1200 may also include other hardware devices for realizing other additional functions. In addition, one of skill in the art should understand that the above-described computing device 1200 may also include only the devices necessary to realize the embodiments of the present application, and need not include all of the devices shown in FIG. 12.

[0145] Some embodiments provide a computing device cluster. FIG. 13 is a schematic diagram of an architecture of a computing device cluster according to an embodiment of the present application.

[0146] The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0147] As shown in FIG. 13, the cluster of computing devices described includes at least one computing device 1200. The memory 1230 in one or more of the computing devices 1200 in the cluster of computing devices may hold the same instructions for performing the data processing method described in the above embodiment.

[0148] In some possible implementations, some of the instructions for performing the data processing method described in the above embodiments may also be separately held in the memory 1230 of the one or more computing devices 1200 in the cluster of computing devices. In other words, a combination of the one or more computing devices 1200 may jointly execute instructions for performing the data processing method described in the above embodiments.

[0149] It is noted that the memories 1230 in the different computing devices 1200 in the cluster of computing devices may store different instructions for performing some of the functions of the computing devices 1200, respectively. That is, the instructions stored in the memories 1230 in the different computing devices 1200 may realize the functions of one or more of the obtaining unit 1101, the processing unit 1102 or the destroying unit 1103.

[0150] The connection between the clusters of computing devices shown in FIG. 14 may be such that, considering that the data processing method provided in the present applicationrequires processing a large number of data packets, it is contemplated that the functions realized by the processing unit 1102 be performed by the computing device 1200B.

[0151] The functions of computing device 1200A illustrated in FIG. 14 may also be accomplished by multiple computing devices 1200. Similarly, the functions of computing device 1200B may be accomplished by multiple computing devices 1200.

[0152] It may be understood that division into the units in the foregoing apparatus is merely logical function division. Each function may correspond to one functional unit, or two or more functions may be integrated into one functional unit. In actual implementation, all or some of the units may be integrated into one physical entity, or may be distributed in different physical entities. In addition, the foregoing functional units may be implemented in a form of hardware, may be implemented in a form of software, or may be implemented in a form of a combination of hardware and software. Whether a function is performed in a form of hardware or software depends on particular applications and design constraint conditions of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of the present disclosure.

[0153] In an example, a functional unit in any one of the foregoing apparatuses may be configured as one or more integrated circuits for implementing the foregoing methods, for example, one or more application-specific integrated circuits (application-specific integrated circuits, ASICs), one or more central processing units (central processing units, CPUs), one or more microprocessors (microcontroller units, MCUs), one or more digital signal processors (digital signal processors, DSP), one or more field programmable gate arrays (field programmable gate arrays, FPGAs), or a combination of at least two of these integrated circuit forms.

[0154] A processor, a processor system, an application processor, a processor circuit, or a processor core may be collectively referred to as a processor. The processor 1220 may include one or a combination of a central processing unit (CPU), a digital signal processor (DSP), a microprocessor (microprocessor unit, MPU), a microcontroller (microcontroller unit, MCU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an artificial intelligence processor (Al processor), or a neural network processing unit (NPU).

[0155] The memory 1230 may include one or more of the following storage media: a random access memory (RAM), a static random access memory (static RAM, SRAM), a dynamic random access memory (dynamic RAM, DRAM), a phase-change memory (PCM), a resistive random access memory (resistive RAM, ReRAM), a magnetoresistive random access memory (magnetoresistive RAM, MRAM), a ferroelectric random access memory (ferroelectric RAM, FRAM), a cache, a register, a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (erasable programmable ROM, EPROM), a hard disk, and the like. In an example, the computer program instructions used to execute the foregoing embodiments may be stored in a non-volatile memory, for example, at least a part of the memory (e.g., one or more of a ROM, a flash memory, an EPROM, or a hard disk). When the computer runs, a part or all of corresponding computer program instructions may be loaded to a memory that has a higher transmission speed with the processor, for example, at least a part of the memory (for example, one or more of a RAM, an SRAM, a DRAM, a PCM, a RERAM, an MRAM, a FRAM, a cache (cache), or a register), so that the processor executes the computer program instructions to perform the steps in the foregoing method embodiments.

[0156] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) having instructions stored thereon which, when executed by a computing device, cause the computing device to perform the method described in the above embodiments.

[0157] The computer-readable storage may be contained in the apparatus described in the above embodiments. It may also be on its own and not assembled into the device.

[0158] In some embodiments, the computer-readable storage medium is a non-volatile computer-readable storage medium, for example including, but is not limited to: portable computer disks, hard disks, RAM, ROM, electrically erasable programmable read-only memory, portable compact disk read-only memory, optical storage, magnetic storage, or any suitable combination of the above.

[0159] In some embodiments, the computer-readable storage medium is any tangible medium containing or storing a program that may be used or used in combination with a command execution system, apparatus or device.

[0160] Some embodiments of the present disclosure provide a computer program product storing computer instructions (e.g., in a non-transitory computer-readable storage medium)which, when executed by a computing device, cause the computing device to perform the method described in the above embodiments.

[0161] Some embodiments of the present disclosure provide a computer program which, when executed by a computing device, causes the computing device to perform the method described in the above embodiments.

[0162] As used herein, the term “if’ is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event],” depending on the context.

[0163] The use of “adapted to” or “configured to” herein is meant as open and inclusive language that does not foreclose devices adapted to or configured to perform additional tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or value beyond those recited.

[0164] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution, and the order of execution of each process shall be determined by its function and internal logic, and shall not constitute any limitation on the implementation of the embodiments of the present application.

[0165] The foregoing descriptions are merely specific implementations of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or replacements that a person skilled in the art could readily conceive of within the technical scope of the present disclosure shall be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims. Numerous details are described herein to provide a thorough understanding of the example embodiments illustrated in the accompanying drawings. However, some embodiments may be practiced without many of the specific details, and the scope of the claims is only limited by those features and aspects specifically recited in the claims. Furthermore, well-knownprocesses, components, and materials have not necessarily been described in exhaustive detail so as to avoid obscuring pertinent aspects of the embodiments described herein.

Claims

WHAT IS CLAIMED IS:

1. A data processing method based on a dataflow graph, the method comprising: obtaining a sub dataflow graph, wherein the sub dataflow graph is a copy of a portion of the dataflow graph, and the sub dataflow graph comprises one or more first nodes and one or more first edges, the one or more first nodes are used for data processing, and the one or more first edges are channels between the one or more first nodes; and processing data based on the sub dataflow graph.

2. The method of claim 1, wherein processing data based on the sub dataflow graph comprises: creating one or more second edges, wherein the one or more second edges are channels between one or more second nodes in the dataflow graph and at least one of the one or more first nodes, and processing data based on the one or more second edges, the one or more second nodes and the sub dataflow graph.

3. The method of claim 1 or 2, wherein the portion of the dataflow graph indicates transformations defined by a predefined class in the dataflow graph.

4. The method of any of claims 1 to 3, wherein the one or more first nodes are copies of one or more nodes in the portion of the dataflow graph, and the one or more first edges are copies of one or more edges in the portion of the dataflow graph.

5. The method of any of claims 1 to 4, wherein obtaining the sub dataflow graph, comprises: obtaining the sub dataflow graph when the portion of the dataflow graph is invoked.

6. The method of claim 2, further comprising: destroying the sub dataflow graph and the one or more second edges.

7. The method of claim 6, wherein destroying the sub dataflow graph and the one or more second edges, comprises: destroying the one or more first nodes, the one or more first edges and the one or more second edges.

8. The method of claim 7, wherein destroying the one or more first nodes, the one or more first edges and the one or more second edges, comprises: destroying the one or more first edges and the one or more second edges automatically when destroying the one or more first nodes.

9. The method of claim 7 or 8, wherein destroying the one or more first nodes, the one or more first edges and the one or more second edges, comprises: destroying the one or more first nodes in response to one or more node destruction instructions.

10. A computing apparatus, comprising: an obtaining unit configured to obtain a sub dataflow graph, wherein the sub dataflow graph is a copy of a portion of the dataflow graph, and the sub dataflow graph comprises one or more first nodes and one or more first edges, the one or more first nodes are used for data processing, and the one or more first edges are channels between the one or more first nodes; and a processing unit configured to process data based on the sub dataflow graph.

11. The computing apparatus of claim 10, wherein the processing unit is configured to: create one or more second edges, wherein the one or more second edges are channels between one or more second nodes in the dataflow graph and at least one of the one or more first nodes, and process data based on the one or more second edges, the one or more second nodes and the sub dataflow graph.

12. The computing apparatus of claim 10 or 11, wherein the portion of the dataflow graph indicates transformations defined by a predefined class in the dataflow graph.

13. The computing apparatus of any of claims 10 to 12, wherein the one or more first nodes are copies of one or more nodes in the portion of the dataflow graph, and the one or more first edges are copies of one or more edges in the portion of the dataflow graph.

14. The computing apparatus of any of claims 10 to 13, wherein the obtaining unit is configured to obtain the sub dataflow graph when the portion of the dataflow graph is invoked.

15. The computing apparatus of claim 11, further comprising: a destroying unit configured to destroy the sub dataflow graph and the one or more second edges.

16. The computing apparatus of claim 15, wherein the destroying unit is configured to destroy the one or more first nodes, the one or more first edges and the one or more second edges.

17. The computing apparatus of claim 16, wherein the destroying unit is configured to destroy the one or more first edges and the one or more second edges automatically when destroying the one or more first nodes.

18. The computing apparatus of claim 16 or 17, wherein the destroying unit is configured todestroy the one or more first nodes in response to one or more node destruction instructions.

19. A computing device cluster, comprising at least one computing device, wherein the computing device comprises a processor and a memory coupled with the processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke and run the computer program stored in the memory, so that the computing device cluster executes the method according to any of claims 1 to 9.

20. A computer-readable storage medium having instructions stored thereon which, when executed by a computing device, cause the computing device to perform the method of any of claims 1 to 9.

21. A computer program product storing computer instructions which, when executed by a computing device, cause the computing device to perform the method of any of claims 1 to 9.

Images