Node generation method and node generation device based on DAG dependency relationship

Abstract

The present invention provides a node generation method and a node generation device based on a DAG dependency relationship, the method comprising: receiving a node generation rule configured by a user; obtaining an executable task set r according to the node generation rule; obtaining a generative set a according to the executable task set r, wherein the generative set a is a set of nodes passed by a valid path between any two nodes in the executable task set r; generating a node according to the executable task set r and the result set of the generative set a. The present invention can enable the nodes generated under complex upstream and downstream dependency relationships to still maintain a local DAG relationship graph, accurately generate the relationship between upstream and downstream nodes, and can ensure the performance in a scenario with a large number of nodes, and support the generation of a large number of tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a node generation method based on DAG dependency relationship and a node generation device based on DAG dependency relationship. Background Art

[0002] At present, the DAG (Directed Acyclic Graph) dependency task local generation technology is based on parsing the json (JavaScript Object Notation, a lightweight data exchange format) string passed from the front-end page to determine the upstream and downstream relationships between nodes, for example:

[0003] [{"task":a,"children":[{"task":b,"children":[{"task":c,"children":[{"task":d}]}]}]}].

[0004] From the above JSON string, we can parse that task a has a subtask b, task b has a subtask c, and task c also has a task d, so we can deduce the direct relationship between the four tasks: a<-b<-c<-d.

[0005] However, the above parsing method has drawbacks and cannot accurately generate the relationship between upstream and downstream nodes. For example, when a user only needs to generate tasks a, c, and d, according to the current logic, task a has no upstream and downstream relationship with tasks c and d, but is a parallel relationship. However, in fact, task a should be the upstream task of b and c. Moreover, the above parsing method does not support the generation of massive tasks. As the number of tasks increases, the json string will become larger, resulting in poor performance and many other problems, that is, it cannot support the generation of massive nodes. Summary of the invention

[0006] To solve the above technical problems, a first object of the present invention is to propose a node generation method based on DAG dependency.

[0007] The second objective of the present invention is to provide a node generation device based on DAG dependency.

[0008] The technical solution adopted by the present invention is as follows:

[0009] An embodiment of the first aspect of the present invention proposes a node generation method based on DAG dependency, comprising the following steps: receiving a node generation rule configured by a user; obtaining an executable task set r according to the node generation rule; obtaining a generable set a according to the executable task set r, wherein the generable set a is a set of nodes passed by a valid path between any two nodes in the executable task set r; and generating a node according to a result set of the executable task set r and the generable set a.

[0010] The node generation method based on DAG dependency relationship proposed in the present invention may also have the following additional technical features:

[0011] According to an embodiment of the present invention, nodes are generated using a multi-threaded approach based on the executable task set r and the result set of the generable task set a.

[0012] According to an embodiment of the present invention, it further includes: setting executable task identifiers for nodes in the executable task set r, and directly configuring the complement of the executable task set r under the generable set a successfully.

[0013] According to an embodiment of the present invention, receiving the node generation rule configured by the user specifically includes: the front-end page receives the node generation rule selected by the user; the front-end page encapsulates the data of the node generation rule into a json data format and transmits it to the back-end.

[0014] An embodiment of the second aspect of the present invention proposes a node generation device based on DAG dependency, including: a receiving module, the receiving module is used to receive a node generation rule configured by a user; a first acquisition module, the first acquisition module is used to obtain an executable task set r according to the node generation rule; a second acquisition module, the second acquisition module is used to obtain a generateable set a according to the executable task set r, wherein the generateable set a is a set of nodes passed by a valid path between any two nodes in the executable task set r; a generation module, the generation module is used to generate a node according to the executable task set r and the result set of the generateable set a.

[0015] The present invention proposed above may also have the following additional technical features:

[0016] According to an embodiment of the present invention, the generation module is specifically used to: generate nodes in a multi-threaded manner according to the result set of the executable task set r and the generable set a.

[0017] According to an embodiment of the present invention, it further includes: a configuration module, which is used to set executable task identifiers for the nodes in the executable task set r, and directly configure the complement of the executable task set r under the generable set a successfully.

[0018] According to an embodiment of the present invention, the receiving module is specifically used for: the front-end page receives the node generation rule selected by the user; the front-end page encapsulates the data of the node generation rule into a json data format and transmits it to the back-end.

[0019] Beneficial effects of the present invention:

[0020] The present invention can enable nodes generated under complex upstream and downstream dependencies to still maintain a local DAG relationship graph, accurately generate the relationship between upstream and downstream nodes, and ensure performance in massive node scenarios, supporting massive task generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a node generation method based on DAG dependency according to an embodiment of the present invention;

[0022] Figure 2 is a node schematic diagram of a DAG graph according to a specific example of the present invention;

[0023] Figure 3 It is a structural diagram of a node generation device based on DAG dependency according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Figure 1 is a flowchart of a node generation method based on DAG dependency according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0026] S1, receiving the node generation rules configured by the user.

[0027] Further, according to an embodiment of the present invention, receiving the node generation rule configured by the user specifically includes: the front-end page receives the node generation rule selected by the user; the front-end page encapsulates the data of the node generation rule into a json data format and transmits it to the back-end.

[0028] Specifically, the user enters the front-end page, selects a node generation rule, and submits the request. The front-end will encapsulate the requested data into JSON data format and transmit it to the back-end. To better illustrate the problem, Figure 2 In the DAG graph shown (this graph is an ideal DAG graph, the actual situation is much more complicated than this, this group of data is only used to illustrate the problem), nodes 1, 6, 9 belong to group a, and nodes 2, 3, 4, 5, 7, 8 belong to group b. Take two cases as examples:

[0029] The user selects batch node generation, selects nodes 1, 6, and 9 to generate node instances, and the front-end encapsulation parameters are as follows (the encapsulated data is screened, and only the necessary fields are displayed here. The actual number of JSON fields is far more than these):

[0030] {"fillDataInfo":{"taskIds":[{"taskId":1},{"taskId":6},{"taskId":9}],"fillDataType":0}}. The taskIds field indicates the selected node, and fillDataType indicates the selected type. Its value of 0 indicates batch generation of node instances.

[0031] The user selects group node generation and selects group a for node generation. The parameters of the front-end package are as follows:

[0032] {"fillDataInfo":{"projects":[{"projectId":a}]},"fillDataType":1} where projects represents the selected group, fillDataType represents the selected type, and its value is 1, indicating that node instances are generated by group.

[0033] After receiving the request, the backend parses the json data passed by the frontend. After passing some necessary checks, the operation record is persisted, and a thread is started to asynchronously execute the node generation operation. Then the frontend is responded to. After receiving the response, the frontend jumps to the node display page. Therefore, through the visual interface, it helps users configure the rules for generating nodes, supports batch selection of any node generation, and can also generate nodes by selecting any group.

[0034] S2, obtain the executable task set r according to the node generation rule.

[0035] Specifically, the elements stored in the executable task set r are tasks that need to be executed. For example, if the node generation rules configured by the user are 1, 6, and 9, then r={1, 6, 9}.

[0036] S3, obtaining a generable set a according to the executable task set r, wherein the generable set a is a set of nodes passed by a valid path between any two nodes in the executable task set r.

[0037] Specifically, Figure 2 As shown, if r = {1,6,9}, then the valid paths between any two nodes among nodes 1, 6, and 9 include: 1-2-3-7-9, 1-4-5-7-9, and 1-4-6. The nodes passed by these three paths are 1, 2, 3, 4, 5, 6, 7, and 9, so the generated set a is {1,2,3,4,5,6,7,9}.

[0038] It is understood that a valid path refers to all paths between any two nodes. For example, Figure 2 As shown, there should be two valid paths between node 1 and node 9, namely 1-2-3-7-9 and 1-4-5-7-9.

[0039] Specifically, the method for generating a valid path according to the elements of the executable task set r may include:

[0040] The elements of the executable task set r may or may not have upstream and downstream dependencies. First, randomly take an element from the r set, take the element as the starting point, query all upstream tasks of the node, and then determine whether these upstream tasks are elements in the r set. If so, the path formed by the node and the starting point is a valid path. Otherwise, take the node as the starting point and repeat the steps just now. Until the location of the upstream task cannot be found. This behavior is called recursion in the program (that is, the method calls itself), and the termination condition of the recursion is that the last node has no upstream task, that is, 1 node. When the randomly taken node recursively completes once, if there are still elements in the r set that have not been recursed, it is necessary to continue to take out the non-recursive elements and continue the recursion.

[0041] S4, generating a node according to the result set of the executable task set r and the generative set a.

[0042] Further, according to an embodiment of the present invention, nodes are generated in a multi-threaded manner according to the result set of the executable task set r and the generative set a.

[0043] The result set of the executable task set r is the elements in the executable task set r, and the result set of the generative set a is the elements in the generative set a.

[0044] Specifically, to generate a node based on the result set of the executable task set r and the generative set a, that is, to start producing node instances, the generative set a needs to be cut into multiple parts and evenly distributed to each thread. When each thread generates an instance, it needs to determine whether an element belongs to the executable task set r. If it does, a flag needs to be set when storing the database to indicate that the task is a runnable task. When each thread completes its work, it updates the database data to indicate that the operation status is completed.

[0045] According to an embodiment of the present invention, the above-mentioned node generation method based on DAG dependency may also include: setting the nodes in the executable task set r as executable task identifiers, and directly configuring the complement of the executable task set r under the generable set a successfully.

[0046] Specifically, the result set of the generable set a is obtained through the executable task set r, and the nodes are generated in a multi-threaded manner. The nodes in the executable task set r will be submitted to the computing engine for calculation. The complement of the set r under the set a will not be submitted to the computing engine for calculation, but will be directly set to success. (Complement: The set composed of elements belonging to the whole set U but not to the set A is called the complement of the set A, denoted as CuA, that is, CuA = {x|x∈U, and x does not belong to A}). In this way, the nodes generated under complex upstream and downstream dependencies can still maintain the local DAG relationship graph, accurately generate the relationship between upstream and downstream nodes, and ensure the performance in massive node scenarios, supporting massive task generation.

[0047] In summary, according to the node generation method based on DAG dependency relationship of the embodiment of the present invention, the node generation rule configured by the user is received; the executable task set r is obtained according to the node generation rule, and the generative set a is obtained according to the executable task set r, wherein the generative set a is the set of nodes passed by the valid path between any two nodes in the executable task set r, and the node is generated according to the result set of the executable task set r and the generative set a. In this way, the nodes generated under the complex upstream and downstream dependency relationship can still maintain the local DAG relationship diagram, accurately generate the relationship between the upstream and downstream nodes, and ensure the performance in the scenario of massive nodes, and support the generation of massive tasks.

[0048] Corresponding to the above-mentioned node generation method based on DAG dependency, the present invention also proposes a node generation device based on DAG dependency. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, the details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0049] Figure 3FIG. 1 is a schematic diagram of a node generation device based on a DAG dependency relationship according to an embodiment of the present invention. Figure 3 As shown, the device includes: a receiving module 10, a first obtaining module 20, a second obtaining module 30, and a generating module 40.

[0050] Among them, the receiving module 10 is used to receive the node generation rules configured by the user; the first acquisition module 20 is used to obtain the executable task set r according to the node generation rules; the second acquisition module 30 is used to obtain the generateable set a according to the executable task set r, wherein the generateable set a is the set of nodes passed by the valid path between any two nodes in the executable task set r; the generating module 40 is used to generate nodes according to the result set of the executable task set r and the generateable set a.

[0051] According to an embodiment of the present invention, the generation module 40 is specifically used to generate nodes in a multi-threaded manner according to the result set of the executable task set r and the generative set a.

[0052] According to one embodiment of the present invention, the above-mentioned node generation device based on DAG dependency also includes: a configuration module, which is used to set the nodes in the executable task set r as executable task identifiers, and directly configure the complement of the executable task set r under the generative set a successfully.

[0053] According to one embodiment of the present invention, the above-mentioned node generation device based on DAG dependency relationship also includes: the receiving module 10 is specifically used for: the front-end page receives the node generation rule selected by the user; the front-end page encapsulates the data of the node generation rule into json data format and transmits it to the back-end.

[0054] According to the node generation device based on DAG dependency relationship of the embodiment of the present invention, the node generation rule configured by the user is received through the receiving module, the first acquisition module acquires the executable task set r according to the node generation rule, the second acquisition module acquires the generative set a according to the executable task set r, wherein the generative set a is the set of nodes passed by the valid path between any two nodes in the executable task set r, and the generation module generates nodes according to the result set of the executable task set r and the generative set a. The device can make the nodes generated under the complex upstream and downstream dependency relationship still maintain the local DAG relationship diagram, accurately generate the relationship between the upstream and downstream nodes, and can ensure the performance in the scenario of massive nodes, and support the generation of massive tasks.

[0055] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. "Multiple" means two or more, unless otherwise clearly and specifically defined.

[0056] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, the technical personnel in this field can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and associate different embodiments or examples and features of different embodiments or examples described in this specification without mutual contradiction.

[0057] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0058] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0059] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0060] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0061] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0062] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0063] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A node generation method based on DAG dependency, characterized in that: The following steps are involved: Receive node generation rules configured by the user; Obtain an executable task set r according to the node generation rule; Acquire a generable set a according to the executable task set r, wherein the generable set a is a set of nodes passed by a valid path between any two nodes in the executable task set r; Generate nodes using a multithreaded manner according to the executable task set r and the result set of the generative set a, wherein the result set of the executable task set r is the elements in the executable task set r, and the result set of the generative set a is the elements in the generative set a; The generating of nodes using a multithreaded manner according to the executable task set r and the result set of the generable set a specifically includes: The generable set a is cut into multiple parts and evenly distributed to each thread; when each thread generates a node, it determines whether an element belongs to the executable task set r. If it does, a flag is set when storing the database to identify the task as an executable task; when each thread completes the work, the database data is updated to indicate that the operation status is completed; The nodes in the executable task set r are set with executable task identifiers, and the complement of the executable task set r under the generable set a is directly configured successfully.

2. The node generation method based on DAG dependency according to claim 1, characterized in that: Receive the node generation rules configured by the user, including: The front-end page receives the node generation rule selected by the user; The front-end page encapsulates the data of the node generation rule into a json data format and transmits it to the back-end.

3. A node generation device based on DAG dependency, characterized in that: include: A receiving module, used for receiving node generation rules configured by a user; A first acquisition module, used to acquire an executable task set r according to the node generation rule; A second acquisition module is used to acquire a generable set a according to the executable task set r, wherein the generable set a is a set of nodes passed by a valid path between any two nodes in the executable task set r; A generation module, used for generating nodes in a multi-threaded manner according to the executable task set r and the result set of the generative set a, wherein the result set of the executable task set r is the elements in the executable task set r, and the result set of the generative set a is the elements in the generative set a; The generation module is specifically used to: cut the generable set a into multiple parts and evenly distribute them to each thread; when each thread generates a node, it determines whether an element belongs to the executable task set r, and if so, sets a flag when storing the database to identify the task as an executable task; when each thread completes the work, it updates the database data to indicate that the operation status is completed; The configuration module is used to set executable task identifiers for the nodes in the executable task set r, and directly configure the complement of the executable task set r under the generative set a successfully.

4. The node generation device based on DAG dependency relationship according to claim 3, characterized in that: The receiving module is specifically used for: The front-end page receives the node generation rule selected by the user; The front-end page encapsulates the data of the node generation rule into a json data format and transmits it to the back-end.

Images