Fusion scheduling method and device based on graph data lineage in full load scenario

Abstract

The present invention provides a fusion scheduling method and device based on graph data lineage in a full-load scenario, the method comprising: when a source table triggers an update, querying the sub-tables of the source table level by level, treating the multi-level sub-tables triggered this time as the first task batch; obtaining at least one first task batch; performing a first rule check on the sub-tables in the same first task batch before scheduling execution, and the sub-tables that pass the first rule check will be scheduled for execution; for tasks in the pre-execution queue, according to the configuration rules, the N first task batches in the queue are merged to obtain a second task batch; after the task batches in the queue to be executed are completed, the second task batch is pulled for execution. The fusion scheduling method and device based on graph data lineage in a full-load scenario provided by the present invention optimize the task scheduling execution strategy according to lineage information, reduce repeated execution, and ensure the smooth operation of more tasks under limited resources.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, and in particular to a fusion scheduling method, device, equipment and computer-readable storage medium based on graph data lineage in a full-load scenario. Background Art

[0002] The description of the background technology in the present invention belongs to the related technology related to the present invention, which is only used to illustrate and facilitate the understanding of the invention content of the present invention, and should not be understood as the applicant explicitly believes or inferred that the applicant believes that it is the prior art of the present invention on the filing date of the first application.

[0003] In recent years, with the development of data middle platforms and the increase in application scenarios, the blood relationship between data has become more and more complicated. Often, when a source table is updated, it will trigger multiple levels of sub-tables, and there will be blood relationships between sub-tables. If no optimization is done, it is very likely that a sub-table will be triggered multiple times, resulting in a waste of computing resources; at the same time, when two or more source tables are updated, there may also be blood relationships between sub-tables triggered by different source tables. In this case, it is very likely that the sub-tables will affect each other, either the sub-table will not trigger the update, or the sub-table will be triggered for multiple updates. When the scheduling is at full load, the linkage update method will cause a large number of sub-table tasks to be triggered, and there is a high probability that a task will be repeatedly called in a short period of time, but the data generated by this task will eventually be based on the last task, so the execution of the intermediate tasks becomes unnecessary, and it also occupies computing resources, which makes the effective utilization of resources low, and the scheduling and execution of tasks become low. In other words, in the prior art, common blood scheduling will appear in the full load scenario, and the old tasks have not been completed and new tasks are produced again.

[0004] In order to solve the above technical problems, the present invention proposes a fusion scheduling method and device based on graph data lineage in a full-load scenario. According to the lineage information, the task scheduling execution strategy is optimized, repeated execution is reduced, and the smooth operation of more tasks under limited resources is ensured. Summary of the invention

[0005] The present invention provides a fusion scheduling method and device based on graph data lineage in a full-load scenario, which optimizes the task scheduling execution strategy according to the lineage information, reduces repeated execution, and ensures the smooth operation of more tasks under limited resources.

[0006] An embodiment of the first aspect of the present invention provides a fusion scheduling method based on graph data lineage in a full-load scenario, including the following steps: when a source table triggers an update, the sub-tables of the source table are queried level by level, and the multi-level sub-tables triggered this time are regarded as the first task batch; at least one first task batch is obtained, and the tasks of the same first task batch are placed in a pre-execution queue; the sub-tables in the same first task batch are subjected to a first rule check before scheduling execution, and the sub-tables that pass the first rule check will be scheduled for execution; the tasks in the pre-execution queue are merged according to the configuration rules, and the N first task batches in the queue are obtained to obtain a second task batch, and the second task batch is placed at the front of the pre-execution queue; wherein N≥2; after the task batches in the queue to be executed are completed, the front task batch is pulled from the pre-execution queue for execution.

[0007] Preferably, in the process of constructing at least one first task batch, the sub-table is subjected to a second rule check, and the sub-table that passes the second rule check is placed in the corresponding first task batch.

[0008] Preferably, the second rule check includes: determining whether to wait for all parent tables to be updated; and / or whether the child table is updated periodically.

[0009] Preferably, the tasks in the pre-execution queue merge the N first task batches in the queue according to the configuration rules to obtain the second task batch. The second task batch is placed in the front step of the pre-execution queue, and the front N first task batches in the queue are merged.

[0010] Preferably, the tasks in the pre-execution queue merge the first N first task batches in the queue according to the configuration rules to obtain the step of the second task batch, which specifically includes the following processes: according to the parameter settings, take N first task batches that meet the requirements from the pre-execution queue; deduplicate the sub-tables involved in the N first task batches; reorganize the blood relationship to generate the second task batch, and put the second task batch at the front of the pre-execution queue.

[0011] Preferably, the parameter setting includes how many times each first task batch can be merged, and / or an upper limit on the number of first task batches that can be merged each time.

[0012] Preferably, in the step of deduplicating the sub-tables involved in the N first task batches, the blood relationship between all the sub-tables is sorted out. If there are identical sub-tables, only one sub-table is retained, and the parent table and sub-table of the table between different first task batches are reconstructed during the deduplication process.

[0013] An embodiment of the second aspect of the present invention also provides a fusion scheduling device based on graph data lineage in a full-load scenario, including: a query module, the query module is configured to query the sub-tables of the source table level by level when the source table triggers an update, and regard the multi-level sub-tables triggered this time as the first task batch; a first task queue module, the first task queue module is configured to obtain at least one first task batch, and the tasks of the same first task batch are placed in a pre-execution queue; a first rule verification module, the first rule verification module is configured to perform a first rule verification on the sub-tables in the same first task batch before scheduling execution, and the sub-tables that pass the first rule verification will be scheduled for execution; a second task queue module, the second task queue module is configured to merge the N first task batches in the queue according to the configuration rules for the tasks in the pre-execution queue to obtain a second task batch, and the second task batch is placed at the front of the pre-execution queue; wherein N≥2; an execution module, the execution module is configured to pull the front task batch from the pre-execution queue for execution after the task batches in the queue to be executed are completed.

[0014] An embodiment of the third aspect of the present invention also provides a fusion scheduling device based on graph data lineage under a full-load scenario, which includes a memory and a processor; wherein the memory is used to store executable program code; the processor is used to read the executable program code stored in the memory to execute a fusion scheduling method based on graph data lineage under a full-load scenario.

[0015] An embodiment of the fourth aspect of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a fusion scheduling method based on graph data lineage is implemented under a full-load scenario.

[0016] The present invention provides a fusion scheduling method, device, equipment and computer-readable storage medium based on graph data lineage in a full-load scenario, which optimizes the task scheduling execution strategy according to lineage information, reduces repeated execution, and ensures the smooth operation of more tasks under limited resources.

[0017] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0019] Figure 1 A flowchart of a fusion scheduling method based on graph data lineage in a full load scenario according to an embodiment of the present invention is shown;

[0020] Figure 2 This is a schematic diagram of a single batch scheduling;

[0021] Figure 3 Schematic diagram of scheduling different batches with blood relationship between batches;

[0022] Figure 4 Schematic diagram of scheduling different batches without blood relationship between batches;

[0023] Figure 5 It is a structural diagram of an embodiment of a fusion scheduling device based on graph data lineage in a full-load scenario of this specification;

[0024] Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium of a fusion scheduling method based on graph data lineage in a full-load scenario of this specification. DETAILED DESCRIPTION

[0025] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0027] The following discussion provides multiple embodiments of the present invention. Although each embodiment represents a single combination of the invention, different embodiments of the present invention can be replaced or combined, so the present invention can also be considered to include all possible combinations of the same and / or different embodiments described. Therefore, if one embodiment includes A, B, C, and another embodiment includes a combination of B and D, then the present invention should also be considered to include embodiments containing one or more of A, B, C, D and all other possible combinations, even though the embodiment may not be explicitly described in the following text.

[0028] Figure 1 A flow chart of a fusion scheduling method based on graph data lineage in a full load scenario according to an embodiment of the present invention is shown; Figure 1 As shown in the figure, the fusion scheduling method based on graph data lineage in the full load scenario includes the following steps:

[0029] Step S01, when the source table triggers an update, the sub-tables of the source table are queried level by level, and the multi-level sub-tables triggered this time are regarded as the first task batch;

[0030] Step S02, obtaining at least one first task batch, and placing tasks of the same first task batch into a pre-execution queue;

[0031] Step S03, performing a first rule check on the sub-tables in the same first task batch before scheduling execution, and the sub-tables that pass the first rule check will be scheduled for execution;

[0032] Step S04, the tasks in the pre-execution queue merge N first task batches in the queue according to the configuration rule to obtain a second task batch, and the second task batch is placed at the front of the pre-execution queue; wherein N≥2;

[0033] Step S05: After the task batches in the to-be-executed queue are executed, the first task batch is pulled from the pre-execution queue for execution.

[0034] Among them, the sub-table in a batch will be checked for rules again before scheduling execution, because its parent table status or other status has changed, and only the table that passes the rule check will be truly scheduled for execution. The embodiment of the present invention mainly optimizes the lineage scheduling of the task scheduling system under full load and implements the fusion scheduling method and system between different task batches, so as to improve the efficiency of task scheduling under full load. In view of the problems of lineage scheduling and task scheduling between different batches, the embodiment of the present invention realizes the optimization of tasks within the same batch, realizes that the same sub-table in the same batch will only be scheduled once, and adjacent different batches are merged into a large batch. The scheduling logic in the same batch is optimized to ensure that the sub-table in the batch will be scheduled only once, which greatly reduces the waste of computing resources; through the optimization of multi-batch fusion, the same table between adjacent batches is avoided from being executed multiple times, which reduces the waste of computing resources; at the same time, it eliminates the abnormal situations such as multiple executions, mutual waiting, skipped execution, etc. caused by the mutual influence of multiple sub-tables between multiple batches when there is a lineage relationship.

[0035] In the fusion scheduling method based on graph data lineage in a full-load scenario provided by an embodiment of the present invention, a second rule check is performed on the child table during the process of constructing at least one first task batch, and the child table that passes the second rule check is placed in the corresponding first task batch. The second rule check includes: determining whether to wait for all parent tables to be updated; and / or whether the child table is updated regularly.

[0036] In the fusion scheduling method based on graph data lineage in a full-load scenario provided by an embodiment of the present invention, the tasks in the pre-execution queue are based on the configuration rules, and the N first task batches in the queue are merged to obtain a second task batch. The second task batch is placed in the front step of the pre-execution queue, and the front N first task batches in the queue are merged.

[0037] In the fusion scheduling method based on graph data lineage in a full-load scenario provided by an embodiment of the present invention, the tasks in the pre-execution queue are integrated according to the configuration rules to obtain the second task batch, which specifically includes the following processes: according to the parameter settings, N first task batches that meet the requirements are taken from the pre-execution queue; the sub-tables involved in the N first task batches are deduplicated; the lineage relationship is reorganized to generate a second task batch, and the second task batch is placed at the front of the pre-execution queue.

[0038] In the graph data lineage-based fusion scheduling method under a full-load scenario provided by an embodiment of the present invention, parameter settings include how many times each first task batch can be fused, and / or an upper limit on the number of first task batches that can be fused each time.

[0039] In the fusion scheduling method based on graph data lineage in a full-load scenario provided by an embodiment of the present invention, in the step of deduplicating the sub-tables involved in N first task batches, the lineage relationship between all sub-tables is sorted out. If there are identical sub-tables, only one sub-table is retained. In the deduplication process, the parent table and child table of the table between different first task batches are reconstructed.

[0040] Figure 2 This is a schematic diagram of a single batch scheduling; Figure 3 Schematic diagram of scheduling different batches with blood relationship between batches; Figure 4 Schematic diagram of scheduling different batches without any blood relationship between the batches.

[0041] like Figure 2 As shown in the figure, a is the source table, and b, c, d, e, and f are sub-tables. Among them, b, c, and d are first-level sub-tables, and e and f are second-level sub-tables. There may be multiple levels of sub-tables in the actual batch, which is only explained here. Before optimization, b and c tasks will trigger e once after running, and e will be scheduled twice; after optimization, e will judge the status of the parent table in this running batch before scheduling. Only when b and c are scheduled and run successfully, e will be scheduled.

[0042] like Figure 3As shown in the figure, a, b, c, d, e, and f are one batch, g, d, f, h, i, and j are another batch, and d is a subtable shared by two batches. Before the batches are merged, if two batches are scheduled successively, d may be scheduled twice, and its subtable f will also be scheduled twice; it is also possible that when one batch is scheduled, the table version mark changes. At this time, if another batch is scheduled, d will be skipped, resulting in the data not being the latest, thus affecting all subtable data of table d to be inconsistent with expectations. After the batches are merged, the two batches are recombined into one batch. When merging, table d will update its blood relationship, that is, a and g are both parent tables of d in the current batch. Only after a and g are scheduled, d will be scheduled. This solves the data update anomaly caused by repeated scheduling of table d or status changes due to scheduling of different batches.

[0043] like Figure 4 As shown in the figure, two batches without common subtables will be scheduled in sequence according to the subtable levels after merging. The level where b is located is the level 1 subtable, and its parent table is the source trigger table; the level where e is located is the level 2 subtable, and its parent table contains the highest level 1; and so on, each table is scheduled once according to the level.

[0044] The fusion scheduling device based on graph data lineage in a full-load scenario provided by an embodiment of the present invention includes: a query module, the query module is configured to query the sub-tables of the source table level by level when the source table triggers an update, and regard the multi-level sub-tables triggered this time as the first task batch; a first task queue module, the first task queue module is configured to obtain at least one first task batch, and the tasks of the same first task batch are placed in a pre-execution queue; a first rule verification module, the first rule verification module is configured to perform a first rule verification on the sub-tables in the same first task batch before scheduling execution, and the sub-tables that pass the first rule verification will be scheduled for execution; a second task queue module, the second task queue module is configured to merge the N first task batches in the queue according to the configuration rules for the tasks in the pre-execution queue to obtain a second task batch, and the second task batch is placed at the front of the pre-execution queue; wherein N≥2; an execution module, the execution module is configured to pull the front task batch from the pre-execution queue for execution after the task batches in the queue to be executed are completed.

[0045] Figure 5 This is a structural diagram of an embodiment of a fusion scheduling device based on graph data lineage in a full load scenario of this specification. Figure 5, which shows a schematic diagram of a structure of a graph data lineage-based fusion scheduling 300 suitable for implementing the full-load scenario of the embodiment of the present disclosure. The electronic devices in the embodiment of the present disclosure may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0046] like Figure 5 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0047] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 2 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0048] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0049] Figure 6 This is a structural diagram of an embodiment of a computer-readable storage medium of a fusion scheduling method based on graph data lineage in a full-load scenario of this specification. Figure 6 As shown, according to the computer-readable storage medium 40 of the embodiment of the present disclosure, non-transitory computer-readable instructions 41 are stored thereon. When the non-transitory computer-readable instructions 41 are executed by the processor, all or part of the steps of the aforementioned fusion scheduling method based on graph data lineage in the full load scenario of each embodiment of the present disclosure are executed.

[0050] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0051] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0052] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: constructs a basic page, and the page code of the basic page is used to build the environment required for the operation of the business page and / or implement the same workflow abstracted from the same business scenario; constructs one or more page templates, and the page templates are used to provide code templates for implementing business functions in business scenarios; based on the corresponding page templates, the final page code of each page of the business scenario is generated through code conversion of the specific functions of each page of the business scenario; the generated final page code of each page is merged into the page code of the basic page to generate the code of the business page.

[0053] Alternatively, the computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: constructs a basic page, the page code of which is used to build the environment required for the operation of the business page and / or implement the same workflow abstracted from similar business scenarios; constructs one or more page templates, the page templates are used to provide code templates for implementing business functions in business scenarios; based on the corresponding page templates, generates the final page code of each page of the business scenario through code conversion of the specific functions of each page of the business scenario; merges the generated final page code of each page into the page code of the basic page to generate the code of the business page.

[0054] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0055] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0056] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0057] The present invention provides a fusion scheduling method and device based on graph data lineage in a full-load scenario, which optimizes the task scheduling execution strategy according to lineage information, reduces repeated execution, and ensures the smooth operation of more tasks under limited resources.

[0058] In the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise clearly defined. The terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0059] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by the terms "upper" and "lower" etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0060] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0061] The above are only some embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fusion scheduling method based on graph data lineage in a full-load scenario, characterized in that: The steps include: When the source table triggers an update, the sub-tables of the source table are queried level by level, and the multi-level sub-tables triggered this time are regarded as the first task batch; At least one first task batch is obtained, and tasks of the same first task batch are placed in a pre-execution queue; Performing a first rule check on the sub-tables in the same first task batch before scheduling execution, and the sub-tables that pass the first rule check will be scheduled for execution; The tasks in the pre-execution queue merge N first task batches in the queue according to the configuration rule to obtain a second task batch, and the second task batch is placed at the front of the pre-execution queue; wherein N≥2; After the task batches in the execution queue are executed, the first task batch is pulled from the pre-execution queue for execution, wherein the tasks in the pre-execution queue merge the first N first task batches in the queue according to the configuration rules to obtain the second task batch, which specifically includes the following process: According to the parameter setting, N first task batches of the sub-tables that pass the first rule verification are taken from the pre-execution queue; De-duplicate the sub-tables involved in the N first task batches; Sort out the blood relationship between all sub-tables. If there are identical sub-tables, only keep one sub-table. During the deduplication process, reconstruct the parent table and sub-table of the table between different first task batches to generate a second task batch, and put the second task batch at the front of the pre-execution queue.

2. The fusion scheduling method based on graph data lineage in a full load scenario according to claim 1 is characterized in that: In the process of constructing at least one of the first task batches, a second rule check is performed on the sub-table, and the sub-table that passes the second rule check is placed in the corresponding first task batch.

3. The fusion scheduling method based on graph data lineage in a full load scenario according to claim 2 is characterized in that: The second rule verification includes: Determine whether to wait for all parent tables to be updated; and / or, Whether the subtable is updated periodically.

4. The fusion scheduling method based on graph data lineage in a full load scenario according to claim 1 is characterized in that: The tasks in the pre-execution queue merge the N first task batches in the queue according to the configuration rules to obtain a second task batch. The second task batch is placed in the front step of the pre-execution queue and merges the front N first task batches in the queue.

5. The fusion scheduling method based on graph data lineage in a full load scenario according to claim 1 is characterized in that: The parameter settings include how many times each first task batch can be merged, and / or an upper limit on the number of first task batches that can be merged each time.

6. A fusion scheduling device based on graph data lineage in a full-load scenario, characterized in that: include: A query module, wherein the query module is configured to query the sub-tables of the source table level by level when the source table triggers an update, and regard the multi-level sub-tables triggered this time as the first task batch; A first task queue module, wherein the first task queue module is configured to obtain at least one first task batch, and tasks of the same first task batch are placed in a pre-execution queue; A first rule checking module, wherein the first rule checking module is configured to perform a first rule check on the sub-tables in the same first task batch before scheduling execution, and the sub-tables that pass the first rule check will be scheduled for execution; A second task queue module, wherein the second task queue module is configured to merge N first task batches in the queue according to a configuration rule to obtain a second task batch, and the second task batch is placed at the front of the pre-execution queue; wherein N≥2; The execution module is configured to pull the first task batch from the pre-execution queue for execution after the task batch in the execution queue is completed, wherein the second task queue module is specifically used to: According to the parameter setting, N first task batches of the sub-tables that pass the first rule verification are taken from the pre-execution queue; De-duplicate the sub-tables involved in the N first task batches; Sort out the blood relationship between all sub-tables. If there are identical sub-tables, only keep one sub-table. During the deduplication process, reconstruct the parent table and sub-table of the table between different first task batches to generate a second task batch, and put the second task batch at the front of the pre-execution queue.

7. A fusion scheduling device based on graph data lineage in a full-load scenario, comprising a memory and a processor; wherein: The memory is used to store executable program code; The processor is used to read the executable program code stored in the memory to execute the fusion scheduling method based on graph data lineage in a full-load scenario according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the fusion scheduling method based on graph data lineage in a full-load scenario as described in any one of claims 1-5.

Images