Job scheduling method and device

Abstract

The invention discloses a job scheduling method and device, and relates to the technical field of computers. The method comprises the following steps: acquiring a configured time tolerance configuration table; on the basis of the time tolerance configuration table, determining an upstream job node on which the job node to be executed depends; based on the time tolerance configuration table, determining whether the upstream job node meets an execution requirement or not; and when the upstream job node meets the execution requirement, starting the execution of the job node to be executed. In the application, differential configuration is performed according to the dependency type between the job nodes, the time tolerance threshold is configured for weak dependency, and when the upstream job node meets the execution requirement corresponding to the dependency type, the upstream job node with strong dependency is completed. And the upstream job node with weak dependence completes or satisfies the time tolerance threshold, and starts the job node to be executed, so that the overall scheduling efficiency of the system is improved, the timeliness of data processing is improved, and the congestion of the job chain is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a job scheduling method and device. BACKGROUND

[0002] With the rapid development of big data, cloud computing and artificial intelligence technology, the process of enterprise digital transformation is deepening, and the scheduling demand of batch and real-time jobs in information systems is rapidly expanding. The number of scheduled jobs often reaches tens of thousands, and the length and span of the dependency link between jobs have significantly increased. For example, in the fields of finance, e-commerce, logistics, etc., the job chain of data access, processing and export involves complex dependency relationships across dates and job chains. This complexity puts higher requirements on the stability and efficiency of the job scheduling system.

[0003] In the current technology, a common job scheduling method is to record the blood relationship and scheduling dependency relationship between job nodes through a directed acyclic graph (DAG), and to generate a job sequence based on this. Based on the job sequence, the corresponding job nodes are executed in order to obtain the processing result.

[0004] As can be seen, in the job scheduling method in the current technology, the downstream job must wait for the upstream job to complete before it can start, which leads to a decrease in the overall scheduling efficiency of the system and poor timeliness of data processing, and even causes job chain congestion. SUMMARY

[0005] Based on the above problems, the present application provides a job scheduling method and device, which differentiates configuration according to the dependency relationship type between job nodes, and configures a time tolerance threshold for weak dependency. When the upstream job node meets the execution requirement corresponding to its dependency relationship type, the job node to be executed is started, which improves the overall scheduling efficiency of the system and improves the timeliness of data processing, avoiding job chain congestion.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] In a first aspect, the embodiments of the present application provide a job scheduling method, comprising:

[0008] obtaining a configured time tolerance configuration table; wherein the time tolerance configuration table includes metadata of the dependency relationship between job nodes, the metadata of the dependency relationship includes a current job node identifier, an upstream job node identifier and a dependency relationship type, the dependency relationship type includes strong dependency and weak dependency, and when the dependency relationship type is weak dependency, the metadata of the dependency relationship between the job nodes further includes a time tolerance threshold;

[0009] determining the upstream job node depended by the job node to be executed based on the time tolerance configuration table;

[0010] determine whether the upstream job nodes meet execution requirements based on the time tolerance configuration table, wherein the execution requirements are that upstream job nodes with a strong dependency type are completed, and upstream job nodes with a weak dependency type are completed or meet a time tolerance threshold;

[0011] When the upstream job nodes meet the execution requirements, execution of the to-be-executed job node is started.

[0012] In a possible implementation, the determining whether the upstream job nodes meet execution requirements based on the time tolerance configuration table comprises:

[0013] determining whether all the upstream job nodes are completed based on the time tolerance configuration table;

[0014] When it is determined that all the upstream job nodes are completed, it is determined that the upstream job nodes meet the execution requirements.

[0015] In a possible implementation, the method further comprises:

[0016] When it is determined that all the upstream job nodes are not completed, it is determined whether all upstream job nodes with a strong dependency type are completed;

[0017] When it is determined that all upstream job nodes with a strong dependency type are completed, it is determined whether all upstream job nodes with a weak dependency type are completed;

[0018] When it is determined that all upstream job nodes with a weak dependency type are completed, it is determined that the upstream job nodes meet the execution requirements.

[0019] In a possible implementation, the method further comprises:

[0020] When it is determined that all upstream job nodes with a weak dependency type are not completed, time tolerance countdown is performed based on a time tolerance threshold corresponding to the upstream job nodes with a weak dependency type that are not completed;

[0021] When the time tolerance countdown ends or all the upstream job nodes with a weak dependency type are completed, it is determined that the upstream job nodes meet the execution requirements.

[0022] In a possible implementation, the method further comprises:

[0023] When the time tolerance countdown ends, but the upstream job nodes of the unfinished dependency type of weak dependency are not all completed, alarm information is generated to indicate that the upstream job nodes of the unfinished dependency type of weak dependency are abnormal.

[0024] In a possible implementation, the method further includes:

[0025] When the upstream job nodes of the unfinished dependency type of weak dependency are all completed, but the time tolerance countdown has not ended, the weak dependency actual waiting time is recorded and stored in the corresponding job scheduling log.

[0026] In a possible implementation, the method further includes:

[0027] The weak dependency actual waiting time and the alarm information in the job scheduling log are acquired;

[0028] Based on the weak dependency actual waiting time and the alarm information, an actual waiting time distribution is generated;

[0029] Based on the actual waiting time distribution, the time tolerance threshold is dynamically adjusted through an artificial intelligence algorithm, and the dynamically adjusted time tolerance threshold is updated to the time tolerance configuration table.

[0030] In a possible implementation, the dependency type between the job nodes is determined in the following manner:

[0031] When the job nodes involve data processing of first-level data, the dependency type corresponding to the job nodes is strong dependency; wherein the data processing includes data access, data processing and data export;

[0032] When the job nodes involve data processing of second-level data, the dependency type corresponding to the job nodes is weak dependency; wherein the first-level data is more important than the second-level data;

[0033] When the job nodes are cross-date dependency, the dependency type between the job nodes is strong dependency.

[0034] In a possible implementation, the metadata of the dependency further includes: a current job chain identifier, an upstream job chain identifier and a dependency date.

[0035] In a second aspect, an embodiment of the present application provides a job scheduling device, which includes:

[0036] The configuration obtaining module is configured to obtain a configured time tolerance configuration table; wherein, the time tolerance configuration table comprises metadata of a dependency relationship between job nodes, and the metadata of the dependency relationship comprises a current job node identifier, an upstream job node identifier, and a dependency relationship type, wherein the dependency relationship type comprises strong dependency and weak dependency, and when the dependency relationship type is weak dependency, the metadata of the dependency relationship between the job nodes further comprises a time tolerance threshold value;

[0037] The upstream determining module is configured to determine, based on the time tolerance configuration table, an upstream job node on which a job node to be executed depends;

[0038] The execution determining module is configured to determine, based on the time tolerance configuration table, whether the upstream job node meets an execution requirement; wherein, the execution requirement is that an upstream job node with a strong dependency relationship type is completed, and an upstream job node with a weak dependency relationship type is completed or meets a time tolerance threshold value;

[0039] The node execution module is configured to start execution of the job node to be executed when the upstream job node meets the execution requirement.

[0040] Compared with the prior art, the present application has the following beneficial effects: the dependency relationship types between job nodes are differentiated according to different configurations, and a time tolerance threshold value is configured for weak dependency; when an upstream job node meets an execution requirement corresponding to its dependency relationship type, i.e., an upstream job node with a strong dependency relationship type is completed, and an upstream job node with a weak dependency relationship type is completed or meets a time tolerance threshold value, the job node to be executed is started, the overall scheduling efficiency of the system is improved, the timeliness of data processing is improved, and job chain congestion is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 A flowchart of an action scheduling method provided by an embodiment of the present application;

[0043] Figure 2 An example diagram of a job scheduling task provided by an embodiment of the present application;

[0044] Figure 3 A flowchart of determining whether an upstream job node meets an execution requirement provided by an embodiment of the present application;

[0045] Figure 4 A structural schematic diagram of an action scheduling device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] As described above, in the prior art, a common action scheduling mode is to convert a data processing action provided by a service provider into a corresponding directed acyclic graph, the directed acyclic graph recording the action blood relationship and the action scheduling dependency relationship between multiple sub-action nodes in the data processing action; when each action scheduling framework has a processing demand for respective basic data, the action blood relationship and the action scheduling dependency relationship are referred to to determine an action sequence from the directed acyclic graph that adapts to the respective processing demand of each action scheduling framework; each action scheduling framework processes the respective basic data according to the respective action sequence to obtain the respective processing result of each action scheduling framework. That is, in the prior art, the blood relationship and the scheduling dependency relationship between action nodes are recorded by a directed acyclic graph, and an action sequence is generated based on this, and the corresponding action nodes are executed in sequence based on the action sequence to obtain a processing result.

[0047] It can be seen that in the prior art, a downstream action must wait for a completed upstream action to start, that is, only strong dependency relationships are considered, but in actual business scenarios, there is a demand for mixed strong and weak dependency relationships between actions, for example, the processing of important data such as product sales amount needs to strictly follow the strong dependency rule to ensure the accuracy of the result, while the timeliness requirement of customer evaluation type data is low, allowing the downstream action to start when part of the upstream action is not completed. Therefore, the prior art only supports strong dependency relationships and does not distinguish between strong and weak dependency relationships. In a multi-action node complex dependency scenario, the scheduling strategy cannot be dynamically adjusted, which may cause important data delay or non-essential data over-waiting, thereby causing the overall scheduling efficiency of the system to decrease, and the timeliness of data processing to be poor, and even causing action chain blocking.

[0048] The present application provides an action scheduling method, comprising: obtaining a configured time tolerance configuration table; determining an upstream action node depended by a to-be-executed action node based on the time tolerance configuration table; determining whether the upstream action node meets an execution requirement based on the time tolerance configuration table; and starting the execution of the to-be-executed action node when the upstream action node meets the execution requirement. In the present application, the dependency relationship types between action nodes are configured differently, and a time tolerance threshold is configured for weak dependency. When the upstream action node meets the execution requirement corresponding to the dependency relationship type, that is, the upstream action node with a strong dependency relationship is completed, and the upstream action node with a weak dependency relationship is completed or meets the time tolerance threshold, the to-be-executed action node is started, which improves the overall scheduling efficiency of the system and improves the timeliness of data processing and avoids action chain congestion.

[0049] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0050] Embodiment one:

[0051] The application provides a job scheduling method. Figures 1-3 As shown in the figure, the job scheduling method provided by the application comprises the following steps: Figure 1

[0052] S101, obtain a configured time tolerance configuration table.

[0053] The time tolerance configuration table comprises metadata of a dependency relationship between job nodes.

[0054] Metadata is data that describes data, which can describe the content, structure, location, relationship, and lifecycle of data.

[0055] In the embodiments of the present application, the metadata of the dependency relationship comprises a current job node identifier, an upstream job node identifier, and a dependency relationship type.

[0056] A job node is the smallest execution unit in a job scheduling system, representing a specific task or operation. Each job node usually corresponds to an independent executable program, script, or service, which is used to complete specific data processing, calculation, or business logic.

[0057] The current job node identifier is used to uniquely identify the current job node, for example, a job node name, a job node ID, etc.

[0058] The upstream job node is used to uniquely identify the upstream job node on which the current job node depends, for example, a job node name, a job node ID, etc.

[0059] The dependency relationship type comprises strong dependency and weak dependency. Strong dependency means that the current job node must wait for the upstream job node to complete before starting; weak dependency means that the current job node does not have to wait for the upstream job node to complete before starting, and it can start during the process of the upstream job node being operated.

[0060] ​Further, when the dependency type is weak dependency, the metadata of the dependency between the job nodes further comprises a time tolerance threshold.

[0061] The time tolerance threshold refers to the minimum waiting time that the current job node can start when the upstream job node is not completed when the dependency type is weak dependency, that is, when the waiting time meets the time tolerance threshold, the execution of the current job node can be started even if the upstream job node with weak dependency type is not completed (job).

[0062] For the convenience of understanding, the following will be introduced in combination with Figure 2 and Table 1.

[0063] As shown in Figure 1 , the current job scheduling task comprises: job node A, job node B, job node C, job node D and job node D; the starting point of the arrow represents the upstream job node, and the ending point of the arrow represents the downstream job node (i.e. the current job node); the solid line between the job nodes represents that the dependency type between the job nodes is strong dependency, and the dotted line between the job nodes represents that the dependency type between the job nodes is weak dependency.

[0064] As shown in Figure 2 , it can be known that the upstream job node of the job node B is the job node A, and the dependency type between the job node B and the job node A is strong dependency; the upstream job node of the job node C is the job node A, and the dependency type between the job node C and the job node A is strong dependency; the upstream job node of the job node E is the job node B, the job node C and the job node D, the dependency type between the job node E and the job node B is strong dependency, the dependency type between the job node E and the job node C is strong dependency, and the dependency type between the job node E and the job node D is weak dependency.

[0065] According to the job scheduling task example diagram as shown in Figure 2 , the configured time tolerance configuration table is shown in Table 1.

[0066] Table 1

[0067]

[0068] As shown in the time tolerance configuration table in Table 1, sequence number 1: the current job node is identified as job node B, the dependent upstream job node is identified as job node A, and the dependency type is strong dependency; sequence number 2: the current job node is identified as job node C, the dependent upstream job node is identified as job node A, and the dependency type is strong dependency; sequence number 3: the current job node is identified as job node E, the dependent upstream job node is identified as job node B, and the dependency type is strong dependency; sequence number 4: the current job node is identified as job node E, the dependent upstream job node is identified as job node C, and the dependency type is strong dependency; sequence number 5: the current job node is identified as job node E, the dependent upstream job node is identified as job node D, and the dependency type is weak dependency.

[0069] In the embodiment of the application, by defining the time tolerance configuration table, combining the strong and weak dependency types and the time tolerance threshold, the flexible management of the job scheduling system for the complex dependency scenario is realized. That is, the strong and weak dependencies are distinguished to ensure the accuracy of important data processing, and the scheduling efficiency of non-critical data is improved.

[0070] In a possible implementation, the dependency relationship type between the job nodes is determined in the following manner:

[0071] When the job node involves data processing of the first-level data, the dependency relationship type corresponding to the job node is strong dependency. When the job node involves (only) data processing of the second-level data, the dependency relationship type corresponding to the job node is weak dependency; when the job nodes are cross-date dependencies, the dependency relationship type between the job nodes is strong dependency.

[0072] Among them, the first-level data refers to important data, for example: product sales amount, sales status, etc. The second-level data refers to non-important data, for example: customer evaluation, etc. The importance of the first-level data is higher than that of the second-level data, which can also be understood as the importance of the first-level data is much higher than that of the second-level data.

[0073] Among them, the data processing includes: data access, data processing and data export.

[0074] Among them, the cross-date dependency refers to: the start of the current job node depends on the output of the upstream job node of the previous day (or the previous few days). That is, no matter whether the data is first-level data or second-level data, as long as the job nodes are cross-date dependencies, the dependency relationship type between the job nodes is strong dependency.

[0075] Exemplarily, according to the difference of time dimension and job chain structure in the job scheduling scene, the dependency relationship between the job nodes is roughly divided into four categories: dependency between jobs in the same day job chain, dependency between jobs across the same day job chains, dependency between jobs across the date job chain, and dependency between jobs across the date job chains.

[0076] The dependency between jobs in the same day job chain refers to the dependency relationship between job nodes in the same day in the same job chain; the dependency between jobs across the same day job chains refers to the dependency relationship between job nodes in the same day in different job chains; the dependency between jobs across the date job chain refers to the dependency relationship between job nodes across the date (in different days) in the same job chain; and the dependency between jobs across the date job chains refers to the dependency relationship between job nodes across the date (in different days) in different job chains.

[0077] For the two types of dependency relationship, the dependency between jobs in the same day job chain and the dependency between jobs across the same day job chains, if the job node involves data processing of the first level data, the dependency relationship type corresponding to the job node is strong dependency; if the job node involves (only) data processing of the second level data, the dependency relationship type corresponding to the job node is weak dependency.

[0078] For the two types of dependency relationship, the dependency between jobs across the date job chain and the dependency between jobs across the date job chains, no matter whether the job node involves data processing of the first level data or data processing of the second level data, the dependency relationship type corresponding to the job node is strong dependency.

[0079] In a possible implementation, the metadata of the dependency relationship in the time tolerance configuration table further includes: a current job chain name, an upstream job chain name, and a dependency date.

[0080] The job chain is a job scheduling execution chain composed of multiple job nodes according to a specific dependency relationship, and is used to describe a complete business process or data processing logic.

[0081] The current job chain identifier is used to uniquely identify the job chain to which the current job node belongs, for example, a job chain name, a job chain ID, etc.

[0082] The upstream job chain identifier is used to uniquely identify the job chain to which the upstream job node belongs.

[0083] The dependency date refers to the time span of the dependency between the current job node and the upstream job node, including the same day or across the date.

[0084] For the convenience of understanding, another configured time tolerance configuration table will be introduced by way of example in combination with Table 2.

[0085] Table 2

[0086]

[0087]

[0088] As shown in the time tolerance configuration table in Table 1, sequence number 1: the current job chain identifier is Chain-1, the current job node identifier is job node B, the upstream job chain identifier is Chain-1, the upstream job node identifier it depends on is job node A, the dependency date is the current day, and the dependency relationship type is strong dependency; sequence number 2: the current job chain identifier is Chain-1, the current job node identifier is job node C, the upstream job chain identifier is Chain-1, the upstream job node identifier it depends on is job node A, the dependency date is the current day, and the dependency relationship type is strong dependency; sequence number 3: the current job chain identifier is Chain-1, the current job node identifier is job node E, the upstream job chain identifier is Chain-1, the upstream job node identifier it depends on is job node B, the dependency date is the current day, and the dependency relationship type is strong dependency; sequence number 4: the current job chain identifier is Chain-1, the current job node identifier is job node E, the upstream job chain identifier is Chain-1, the upstream job node identifier it depends on is job node C, the dependency date is cross date, and the dependency relationship type is strong dependency; sequence number 5: the current job chain identifier is Chain-1, the current job node identifier is job node E, the upstream job chain identifier is Chain-2, the upstream job node identifier it depends on is job node D, the dependency date is the current day, and the dependency relationship type is weak dependency.

[0089] S102, based on the time tolerance configuration table, determine the upstream job node that the to-be-executed job node depends on.

[0090] Specifically, taking the identifier of the to-be-executed job node as the current job node identifier, the upstream job node it depends on is found from the time tolerance configuration table. For example, taking the time tolerance configuration table as Table 1 and the to-be-executed job node as job node E, the upstream job nodes that the to-be-executed job node depends on are determined to be job node B, job node C, and job node D.

[0091] S103, based on the time tolerance configuration table, determine whether the upstream job node meets the execution requirement.

[0092] Wherein, the execution requirement is: the upstream job node with strong dependency relationship type is completed, and the upstream job node with weak dependency relationship type is completed or meets the time tolerance threshold.

[0093] Specifically, for the upstream job node with the dependency type of strong dependency, the strong dependency execution requirement is that all the upstream job nodes with strong dependency must have completed execution, and the subsequent job node to be executed can be started.

[0094] Specifically, for the upstream job node with the dependency type of weak dependency, the weak dependency execution requirement is divided into three cases: the first case is that all the upstream job nodes with weak dependency have completed execution, and the subsequent job node to be executed can be started; the second case is that part of the upstream job nodes with weak dependency have completed execution (that is, part of the upstream job nodes with weak dependency have not completed execution), but the waiting time meets the time tolerance threshold, and the subsequent job node to be executed can be started; and the third case is that all the upstream job nodes with weak dependency have not completed execution, but the waiting time meets all the time tolerance thresholds, and the subsequent job node to be executed can be started.

[0095] In the embodiment of the present application, when the strong dependency execution requirement is met and the weak dependency execution requirement is met, it is determined that the upstream job node meets the execution requirement.

[0096] When it is determined that the upstream job node does not meet the execution requirement, S103 is repeated until it is determined that the upstream job node meets the execution requirement.

[0097] When it is determined that the upstream job node meets the execution requirement, S104 is performed.

[0098] S104, starting execution of the job node to be executed.

[0099] When the upstream job node meets the execution requirement (strong dependency completion + weak dependency completion or meeting the time tolerance threshold), the role execution interface is called to start execution of the current job node.

[0100] For the convenience of understanding, the following will be described in detail in combination with Figure 3 how to determine whether the upstream job node meets the execution requirement in the embodiment of the present application.

[0101] S301, judging whether all the upstream job nodes have completed based on the time tolerance configuration table.

[0102] Specifically, it is judged whether the upstream job nodes with the dependency type of weak dependency and the upstream job nodes with the dependency type of strong dependency have all completed execution.

[0103] When it is determined that all the upstream job nodes have completed, it is indicated that the upstream job nodes with strong dependency meet the strong dependency execution requirement, and the upstream job nodes with weak dependency meet the weak dependency execution requirement, and then S306 is performed.

[0104] When it is determined that all the upstream job nodes are not completed, S301 is performed.

[0105] S302, it is determined whether all the upstream job nodes of the strong dependency type are completed.

[0106] Specifically, when it is determined that all the upstream job nodes are not completed, it is required to continue to determine whether the upstream job nodes of the strong dependency type meet the strong dependency execution requirement and whether the upstream job nodes of the weak dependency type meet the weak dependency execution requirement. Therefore, firstly, it is determined whether all the upstream job nodes of the strong dependency type are completed, i.e., whether the upstream job nodes of the strong dependency type meet the strong dependency execution requirement.

[0107] When it is determined that all the upstream job nodes of the strong dependency type are not completed, S302 is repeated until all the upstream job nodes of the strong dependency type are completed.

[0108] When it is determined that all the upstream job nodes of the strong dependency type are completed, S303 is performed.

[0109] S303, it is determined whether all the upstream job nodes of the weak dependency type are completed.

[0110] Specifically, when it is determined that all the upstream job nodes of the strong dependency type are completed, i.e., the upstream job nodes of the strong dependency type meet the strong dependency execution requirement, it is continued to determine whether all the upstream job nodes of the weak dependency type are completed, i.e., whether the upstream job nodes of the weak dependency type meet the weak dependency execution requirement.

[0111] When it is determined that all the upstream job nodes of the weak dependency type are completed, it is indicated that the upstream job nodes of the strong dependency type meet the strong dependency execution requirement and the upstream job nodes of the weak dependency type meet the weak dependency execution requirement, and then S306 is performed.

[0112] When it is determined that all the upstream job nodes of the weak dependency type are not completed, S304 is performed.

[0113] S304, time tolerance countdown is performed based on the time tolerance threshold corresponding to the upstream job node of the weak dependency type which is not completed.

[0114] In a possible implementation, the maximum value in the time tolerance threshold corresponding to all weakly dependent upstream job nodes is determined as a target time tolerance threshold based on the time tolerance threshold corresponding to the upstream job node with the uncompleted dependency relationship type being weakly dependent, and the time tolerance countdown is performed based on the target time tolerance threshold. Determining the maximum value in the time tolerance threshold as the target time tolerance threshold ensures that all job nodes with the dependency relationship type being weakly dependent meet the weakly dependent execution requirement, ensures successful job scheduling, and avoids overall scheduling failure caused by a single job node.

[0115] In S305, it is determined whether the time tolerance countdown is ended or all upstream job nodes with the dependency relationship type being weakly dependent are completed.

[0116] When the time tolerance countdown is ended or all upstream job nodes with the dependency relationship type being weakly dependent are completed, it is indicated that all upstream job nodes with the dependency relationship type being weakly dependent are completed, or the time tolerance countdown is ended but part of the upstream job nodes with the dependency relationship type being weakly dependent are completed, or the time tolerance countdown is ended but none of the upstream job nodes with the dependency relationship type being weakly dependent is completed, which indicates that the weakly dependent job nodes meet the weakly dependent execution requirement, and then S306 is performed.

[0117] When the time tolerance countdown is not ended and all upstream job nodes with the dependency relationship type being weakly dependent are not completed, S305 is repeatedly performed until the time tolerance countdown is ended or all upstream job nodes with the dependency relationship type being weakly dependent are completed, and then S306 is performed.

[0118] In a possible implementation, when the time tolerance countdown is ended but all upstream job nodes with the dependency relationship type being weakly dependent are not completed, alarm information is generated to indicate that the upstream job nodes with the dependency relationship type being weakly dependent are abnormal.

[0119] Specifically, when the time tolerance countdown is ended but there are still upstream job nodes with the dependency relationship type being weakly dependent that are not completed, alarm information is generated to prompt the abnormality. The alarm information includes, for example, a list of the upstream job nodes with the dependency relationship type being weakly dependent that are not completed, a timeout timestamp (the specific time when the countdown is ended), and the like.

[0120] In a possible implementation, when all upstream job nodes with the dependency relationship type being weakly dependent that are not completed are completed but the time tolerance countdown is not ended, the actual waiting time of the weakly dependent is recorded and stored in the corresponding job scheduling log.

[0121] Specifically, when all upstream job nodes of the weak dependency type are completed, but the time tolerance countdown has not ended (i.e., the actual waiting time of the weak dependency is less than the time tolerance threshold), the actual waiting time of the weak dependency is recorded and stored in the corresponding job scheduling log.

[0122] The actual waiting time of the weak dependency is the time difference from the start time of the time tolerance countdown to the time when all upstream job nodes of the weak dependency are completed.

[0123] S306, determine whether the upstream job node meets the execution requirement.

[0124] For example, the time tolerance configuration table is Table 1, the job node to be executed is job node E, the upstream job nodes on which the job node to be executed depends are job node B, job node C, and job node D. Among them, the dependency types corresponding to job node B and job node C are strong dependency, the dependency type corresponding to job node D is weak dependency, and the time tolerance threshold is 600 seconds.

[0125] Specifically, the process of determining whether the upstream job node meets the execution requirement is: judging whether job node B and job node C complete execution; when job node B and job node C complete execution, judging whether job node D completes execution; if job node D does not complete execution, performing time tolerance countdown based on the time tolerance threshold of 600 seconds; when job node D completes execution or the time tolerance countdown ends (i.e., waiting for 600 seconds), determining that the upstream job node meets the execution requirement; and starting the execution of job node E.

[0126] In one possible implementation, the actual waiting time of the weak dependency and the alarm information in the job scheduling log are obtained; based on the actual waiting time of the weak dependency and the alarm information, an actual waiting time distribution is generated; based on the actual waiting time distribution, the time tolerance threshold is dynamically adjusted through an artificial intelligence algorithm, and the dynamically adjusted time tolerance threshold is updated to the time tolerance configuration table.

[0127] The job scheduling log includes the actual waiting time of the weak dependency, and the alarm information includes the timeout timestamp.

[0128] Specifically, all weak dependency actual waiting times are extracted from the job scheduling log, and all timeout timestamps are extracted from the alarm information; through statistical analysis, based on the actual waiting time of the weak dependency and the timeout timestamp, an actual waiting time distribution is generated, that is, the probability times of two boundary conditions that the actual waiting time exceeds the "time tolerance threshold" or the actual waiting time is zero are calculated; based on the actual waiting time distribution, the time tolerance threshold is dynamically adjusted through an artificial intelligence algorithm (such as a genetic algorithm), and the dynamically adjusted time tolerance threshold is updated to the time tolerance configuration table.

[0129] In the embodiments of the present application, the threshold is automatically adjusted according to historical data, the problem of over-loose or over-tight caused by a fixed time tolerance threshold is avoided, and the scheduling strategy is optimized in a targeted manner by identifying high-frequency operations with timeout and zero waiting time. Thus, the job chain blockage or resource waste caused by unreasonable threshold is reduced.

[0130] The embodiments of the present application provide a job scheduling method, including: obtaining a configured time tolerance configuration table; determining an upstream job node relied on by a to-be-executed job node based on the time tolerance configuration table; determining whether the upstream job node meets an execution requirement based on the time tolerance configuration table; and starting execution of the to-be-executed job node when the upstream job node meets the execution requirement. In the embodiments of the present application, the dependency relationship types between job nodes are differentiated and configured, and a time tolerance threshold is configured for weak dependency. When the upstream job node meets the execution requirement corresponding to the dependency relationship type (the upstream job node with strong dependency is completed, and the upstream job node with weak dependency is completed or meets the time tolerance threshold), the to-be-executed job node is started, the scheduling efficiency of the whole system is improved, the timeliness of data processing is improved, and job chain congestion is avoided.

[0131] Embodiment two:

[0132] The embodiments of the present application will be described in detail below. Figure 4 The embodiments of the present application provide a job scheduling device.

[0133] As shown in Figure 4 , the job scheduling device provided by the embodiments of the present application includes the following modules:

[0134] The configuration obtaining module 401 is configured to obtain a configured time tolerance configuration table; wherein the time tolerance configuration table includes metadata of dependency relationships between job nodes, the metadata of dependency relationships includes a current job node identifier, an upstream job node identifier, and a dependency relationship type, the dependency relationship type includes strong dependency and weak dependency, and when the dependency relationship type is weak dependency, the metadata of dependency relationships between job nodes further includes a time tolerance threshold;

[0135] The upstream determining module 402 is configured to determine an upstream job node relied on by a to-be-executed job node based on the time tolerance configuration table;

[0136] The execution judging module 403 is configured to determine whether the upstream job node meets an execution requirement based on the time tolerance configuration table; wherein the execution requirement is that the upstream job node with strong dependency is completed, and the upstream job node with weak dependency is completed or meets the time tolerance threshold;

[0137] The node execution module 404 is configured to start execution of the job node to be executed when the upstream job node meets the execution requirement.

[0138] In a possible implementation, the execution judgment module 403 is specifically configured to determine whether all the upstream job nodes are completed based on the time tolerance configuration table; and when it is determined that all the upstream job nodes are completed, it is determined that the upstream job node meets the execution requirement.

[0139] In a possible implementation, the execution judgment module 403 is specifically configured to, when it is determined that all the upstream job nodes are not completed, determine whether all the upstream job nodes of the strong dependency type are completed; when it is determined that all the upstream job nodes of the strong dependency type are completed, determine whether all the upstream job nodes of the weak dependency type are completed; and when it is determined that all the upstream job nodes of the weak dependency type are completed, determine that the upstream job node meets the execution requirement.

[0140] In a possible implementation, the execution judgment module 403 is further configured to, when it is determined that all the upstream job nodes of the weak dependency type are not completed, perform time tolerance countdown based on a time tolerance threshold corresponding to the upstream job node of the weak dependency type that is not completed; and when the time tolerance countdown ends or all the upstream job nodes of the weak dependency type are completed, determine that the upstream job node meets the execution requirement.

[0141] In a possible implementation, the apparatus further includes an alarm generation module configured to, when the time tolerance countdown ends but all the upstream job nodes of the weak dependency type that are not completed, generate alarm information to indicate that the upstream job node of the weak dependency type that is not completed is abnormal.

[0142] In a possible implementation, the apparatus further includes a log recording module configured to, when all the upstream job nodes of the weak dependency type that are not completed are completed but the time tolerance countdown is not ended, record a weak dependency actual waiting time and store the weak dependency actual waiting time into a corresponding job scheduling log.

[0143] In a possible implementation, the apparatus further includes a threshold adjustment module configured to acquire the weak dependency actual waiting time and the alarm information in the job scheduling log; generate an actual waiting time distribution based on the weak dependency actual waiting time and the alarm information; dynamically adjust the time tolerance threshold by using an artificial intelligence algorithm based on the actual waiting time distribution; and update the dynamically adjusted time tolerance threshold into the time tolerance configuration table.

[0144] The embodiment of the application provides a kind of job scheduling device, comprising: configuration acquisition module 401, for obtaining the time tolerance configuration table of configuration;Upstream determination module 402, for determining the upstream job node that the job node to be executed depends on based on time tolerance configuration table;Execution judging module 403, for determining whether upstream job node meets execution requirement based on time tolerance configuration table;Node execution module 404, for when upstream job node meets execution requirement, then start the execution of the job node to be executed.In the embodiment of the application, the dependency relationship type between job nodes is configured differently, and a time tolerance threshold is configured for weak dependency.When upstream job node meets the execution requirement corresponding to its dependency relationship type (upstream job node is completed for strong dependency, and upstream job node is completed or meets time tolerance threshold for weak dependency), the job node to be executed is started, the scheduling efficiency of the whole system is improved, the timeliness of data processing is improved, and job chain congestion is avoided.

[0145] It should be noted that each of the embodiments in the specification adopts a progressive manner for description, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. Especially, for the device embodiment, since it is basically similar to the method embodiment, it is described more simply, and the related parts can be referred to the part of the method embodiment. The device embodiment described above is only schematic, and the unit described as separate component can be or can not be physically separated, and the component prompted as unit can be or can not be physical unit, that is, it can be located in one place, or can be distributed to multiple network units. Part or all modules can be selected according to actual needs to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0146] The above is only a specific embodiment of the application, but the protection scope of the application is not limited to this, any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be limited by the protection scope of claims.

Claims

1. A job scheduling method characterized by comprising: The method comprises: obtaining a configured time tolerance configuration table; wherein the time tolerance configuration table comprises metadata of a dependency relationship between job nodes, and the metadata of the dependency relationship comprises a current job node identifier, an upstream job node identifier, and a dependency relationship type, the dependency relationship type comprising a strong dependency and a weak dependency, and when the dependency relationship type is a weak dependency, the metadata of the dependency relationship between the job nodes further comprises a time tolerance threshold; determining, based on the time tolerance configuration table, an upstream job node on which a to-be-executed job node depends; determining, based on the time tolerance configuration table, whether the upstream job node meets an execution requirement; wherein the execution requirement is that an upstream job node with a strong dependency relationship is completed, and an upstream job node with a weak dependency relationship is completed or meets a time tolerance threshold; when the upstream job node meets the execution requirement, starting execution of the to-be-executed job node.

2. The method of claim 1, wherein, The method further comprises: determining, based on the time tolerance configuration table, whether all the upstream job nodes are completed; when it is determined that all the upstream job nodes are completed, it is determined that the upstream job nodes meet the execution requirement.

3. The method of claim 2, wherein, The method further comprises: when it is determined that all the upstream job nodes are not completed, it is determined whether all upstream job nodes with a strong dependency relationship are completed; when it is determined that all upstream job nodes with a strong dependency relationship are completed, it is determined whether all upstream job nodes with a weak dependency relationship are completed; when it is determined that all upstream job nodes with a weak dependency relationship are completed, it is determined that the upstream job nodes meet the execution requirement.

4. The method of claim 3, wherein, The method further comprises: when it is determined that all upstream job nodes with a weak dependency relationship are not completed, time tolerance countdown is performed based on a time tolerance threshold corresponding to an upstream job node with a weak dependency relationship that is not completed; when the time tolerance countdown ends or all upstream job nodes with a weak dependency relationship are completed, it is determined that the upstream job nodes meet the execution requirement.

5. The method of claim 4, wherein, The method further comprises: when the time tolerance countdown ends but all upstream job nodes with a weak dependency relationship that are not completed, an alarm information is generated to indicate that the upstream job nodes with a weak dependency relationship that are not completed are abnormal.

6. The method of claim 5, wherein, The method further comprises: when all upstream job nodes with a weak dependency relationship that are not completed are completed but the time tolerance countdown is not ended, a weak dependency actual waiting time is recorded and stored in a corresponding job scheduling log.

7. The method of claim 6, wherein, The method further comprises: obtaining the weak dependency actual waiting time and the alarm information in the job scheduling log; generating an actual waiting time distribution based on the weak dependency actual waiting time and the alarm information. Based on the actual waiting time distribution, the time tolerance threshold is dynamically adjusted by an artificial intelligence algorithm, and the dynamically adjusted time tolerance threshold is updated to the time tolerance configuration table.

8. The method of claim 1, wherein, The dependency type between the job nodes is determined in the following manner: When the job nodes involve data processing of first-level data, the dependency type corresponding to the job nodes is strong dependency; wherein the data processing includes data access, data processing and data export; When the job nodes involve data processing of second-level data, the dependency type corresponding to the job nodes is weak dependency; wherein the importance of the first-level data is higher than that of the second-level data; When the job nodes are cross-date dependency, the dependency type between the job nodes is strong dependency.

9. The method according to any one of claims 1 to 8, characterized in that, The metadata of the dependency further includes: current job chain identifier, upstream job chain identifier, and dependency date.

10. A job scheduling apparatus characterized by comprising: Comprise: The configuration acquisition module is used to acquire the configured time tolerance configuration table; wherein the time tolerance configuration table includes: metadata of dependency relationship between job nodes, the metadata of the dependency relationship includes: current job node identifier, upstream job node identifier and dependency type, the dependency type includes: strong dependency and weak dependency, when the dependency type is weak dependency, the metadata of the dependency relationship between the job nodes further includes: time tolerance threshold; The upstream determination module is used to determine the upstream job node depended by the to-be-executed job node based on the time tolerance configuration table; The execution judgment module is used to determine whether the upstream job node meets the execution requirement based on the time tolerance configuration table; wherein the execution requirement is: the upstream job node with strong dependency type is completed, and the upstream job node with weak dependency type is completed or meets the time tolerance threshold; The node execution module is used to start the execution of the to-be-executed job node when the upstream job node meets the execution requirement.

Images