A big data task scheduling method, device, electronic device and storage medium

By dynamically calculating the impact coefficient values ​​and resource conditions of the task, adjusting the priority and allocation of tasks, the problems of low task scheduling efficiency and high operation and maintenance costs in the Quartz framework are solved, and the task scheduling efficiency and reduction of operation and maintenance costs are achieved.

CN118550716BActive Publication Date: 2024-11-01HANGZHOU ROBAM APPLIANCES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411027929.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-01
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The existing task scheduling based on the Quartz framework can only adopt static settings in task priority configuration, and cannot be changed according to time and requirements, resulting in inefficient task scheduling and increased operation and maintenance costs.

Method used

By receiving the task list to be scheduled, obtaining the initial scheduling priority and dependency of the task, calculating the coefficient value of each task's impact on downstream tasks, dynamically adjusting the final scheduling priority, and allocating tasks to the execution server according to the resource situation to achieve dynamic Priority adjustment and resource balance.

Benefits of technology

It improves task scheduling efficiency, ensures stable output time of important tasks, reduces manual manual operations, reduces operation and maintenance costs, and improves the high availability and efficiency of the scheduling system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118550716B_ABST
    Figure CN118550716B_ABST
Patent Text Reader

Abstract

The present application provides a big data task scheduling method, device, electronic device and storage medium, the method comprising: receiving a task list to be scheduled through a task allocation server; obtaining the initial scheduling priority of each task and the dependency relationship between each task in the task list; calculating the influence coefficient value of each task on its downstream task based on the dependency relationship between each task; determining the final scheduling priority based on the initial scheduling priority and the influence coefficient value of each task; and allocating each task to a task execution server based on the final scheduling priority. The present application improves scheduling efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of distributed scheduling technology, and in particular to a big data task scheduling method, device, electronic device and storage medium. Background Art

[0002] With the development of big data, there is an increasing demand for the application and development of big data in equipment such as kitchen appliances. These demands include but are not limited to data collection, data calculation, data loading, etc. The server needs to carry thousands of tasks for scheduling and operation every day.

[0003] Currently, in terms of task priority configuration, task scheduling based on the Quartz framework can only adopt static task priority settings, which cannot change over time and with demand. Priorities can only be adjusted manually, resulting in inefficient task scheduling and increased operation and maintenance costs. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a big data task scheduling method, device, electronic device and storage medium to improve task scheduling efficiency while reducing operation and maintenance costs.

[0005] In a first aspect, a big data task scheduling method is provided, the method comprising:

[0006] Receiving a list of tasks to be scheduled through a task allocation server;

[0007] Obtain the initial scheduling priority of each task and the dependencies between tasks in the task list;

[0008] Calculate the impact coefficient of each task on its downstream tasks based on the dependencies between the tasks;

[0009] Determine the final scheduling priority based on the initial scheduling priority and impact coefficient value of each task;

[0010] Each task is assigned to a task execution server based on the final scheduling priority.

[0011] Optionally, obtaining dependencies between tasks in the task list includes:

[0012] Get the task configuration table of each task in the task list; each task corresponds to a task configuration table;

[0013] Read the relationship between each task configuration table through the preset relationship reading tool;

[0014] Generate a task dependency configuration table based on the relationship between each task configuration table;

[0015] Determine the dependencies between tasks based on the task dependency configuration table.

[0016] Optionally, calculating the influence coefficient value of each task on its downstream tasks based on the dependency relationship between the tasks includes:

[0017] Based on the dependencies between tasks, obtain the current number of downstream tasks for each task, the maximum number of downstream tasks among all tasks, and the minimum number of downstream tasks;

[0018] The influence coefficient value of each task on its downstream tasks is calculated based on the current number of downstream tasks, the maximum number of downstream tasks, the minimum number of downstream tasks and the preset influence coefficient calculation rules.

[0019] Optionally, determining the final scheduling priority based on the initial scheduling priority and the influence coefficient value of each task includes:

[0020] The initial scheduling priority is adjusted with the value of the influence coefficient as an adjustment parameter to obtain a final scheduling priority.

[0021] Optionally, a plurality of scheduling queues are provided in the task execution server, each scheduling queue corresponds to a task type and each scheduling queue is provided with a concurrency threshold; and allocating each task to the task execution server based on the final scheduling priority includes:

[0022] Obtain the task type of each task in the task list, the number of tasks of each task type, and the remaining resources in the task execution server;

[0023] If the remaining resources in the task execution server are greater than the preset resource threshold, the concurrency threshold of each scheduling queue in the task execution server is obtained;

[0024] Determine whether the number of tasks of each task type exceeds the concurrency threshold in the corresponding scheduling queue;

[0025] If it exceeds, the task is assigned to the task execution server based on the final scheduling priority.

[0026] Optionally, the task allocation server includes a primary task allocation server and a backup task allocation server, and receiving the task list to be scheduled through the task allocation server includes:

[0027] Monitor the operation status of the main task allocation server in real time through the backup task allocation server; the operation status includes at least normal and fault;

[0028] If the running status is normal, the task list to be scheduled is received through the main task allocation server;

[0029] If the operating status is a failure, the backup task allocation server is switched to receive the task list to be scheduled.

[0030] Optionally, the method further comprises:

[0031] Real-time monitoring of the operation data of the task execution server;

[0032] Determine a task running error type based on the running data, where the task running error type includes a specified error type and a non-specified error type;

[0033] Automatically retry repair based on specified error types;

[0034] Generates remote warning information based on unspecified error types.

[0035] In a second aspect, a big data task scheduling device is provided, the device comprising:

[0036] A receiving unit, used for receiving a list of tasks to be scheduled through a task allocation server;

[0037] An acquisition unit, used to acquire the initial scheduling priority of each task and the dependency relationship between the tasks in the task list;

[0038] A calculation unit, used to calculate the influence coefficient value of each task on its downstream tasks based on the dependency relationship between the tasks;

[0039] A determination unit, configured to determine a final scheduling priority based on an initial scheduling priority and an influence coefficient value of each task;

[0040] The allocation unit is used to allocate each task to the task execution server based on the final scheduling priority.

[0041] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0042] Memory, used to store computer programs;

[0043] The processor is used to implement any method step described in the first aspect when executing a program stored in the memory.

[0044] According to a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0045] The embodiment of the present invention provides a big data task scheduling method, device, electronic device and storage medium, which obtains the initial scheduling priority of each task and the dependency relationship between each task in the task list; calculates the influence coefficient value of each task on its downstream tasks based on the dependency relationship between each task; and determines the final scheduling priority based on the initial scheduling priority and influence coefficient value of each task. It realizes the dynamic adjustment of scheduling priority according to the dependency relationship and influence degree between different tasks, which can ensure that important tasks will be scheduled first, ensure the stability of the output time of important tasks, and also releases tedious manual operations, ensuring that important tasks can be completed in time and improving scheduling efficiency.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flowchart of a big data task scheduling method provided by an embodiment of the present invention is shown;

[0049] Figure 2 A flowchart for determining the dependency relationship between tasks provided by an embodiment of the present invention is shown;

[0050] Figure 3 A schematic diagram of the structure of a big data task scheduling device provided by an embodiment of the present invention is shown;

[0051] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0053] Considering that the current task scheduling based on the Quartz framework can only use static task priority settings in terms of task priority configuration, which cannot change over time and demand, and can only be adjusted manually and continuously, resulting in low task scheduling efficiency and increased operation and maintenance costs. Based on this, an embodiment of the present invention provides a big data task scheduling method and device, which is described below through an embodiment.

[0054] In a distributed task scheduling system, generally multiple workers are deployed under a master. The master acts as a task allocation server to schedule and allocate tasks, and the worker acts as a task execution server to execute tasks.

[0055] The embodiment of the present invention provides a big data task scheduling method, which is applied to a task allocation server, such as Figure 1 As shown, the method comprises the following steps:

[0056] Step S101: receiving a list of tasks to be scheduled through a task allocation server.

[0057] The task list includes at least the task name, task type, and task quantity.

[0058] Step S102: Obtain the initial scheduling priority of each task and the dependency relationship between the tasks in the task list.

[0059] In the embodiment of the present application, the initial scheduling priority is configured based on the priority function supported by Quartz itself, and can be preset according to the task type and importance of each task. In an example, the setting rules are as shown in Table 1 below:

[0060] Table 1: Initial scheduling priority configuration rules

[0061]

[0062] In Table 1, the initial scheduling priority is divided into five levels: highest level, very important, important, normal and below normal, and the default priority value of the highest level is 100, and the default values ​​of the priorities of the remaining levels differ by 10.

[0063] The details of obtaining the dependency relationship between the tasks will be described in detail in the following embodiments and will not be repeated here.

[0064] Step S103: Calculate the influence coefficient of each task on its downstream tasks based on the dependency relationship between the tasks.

[0065] In the embodiment of the present application, the impact coefficient value of each task is proportional to the number of its downstream tasks, that is, for a task, the more downstream tasks it has, the greater its global impact coefficient value.

[0066] Step S104: Determine the final scheduling priority based on the initial scheduling priority and the impact coefficient value of each task.

[0067] In the embodiment of the present application, the impact coefficient value of each task is related to the number of downstream tasks. Since the tasks to be scheduled are constantly changing, the dependencies between tasks are also dynamically changing, and their impact coefficient values ​​are also constantly changing. Therefore, based on the priority configuration supported by Quartz itself, by combining the impact coefficient value to dynamically adjust the scheduling priority, it can ensure that important tasks will be scheduled first, ensure the stability of the output time of important tasks, and also release tedious manual operations, ensure that important tasks can be completed in time, and improve scheduling efficiency.

[0068] Step S105: Allocate each task to a task execution server based on the final scheduling priority.

[0069] It is inevitable that there will be competition for cluster resources between the tasks to be scheduled. If there is no priority rule, the fluctuation of output time will be uncontrollable when a large number of tasks are running. Therefore, based on the priority configuration supported by Quartz itself, the embodiment of the present application cleverly utilizes the dynamic dependency relationship between tasks to characterize the degree of influence between tasks, and dynamically adjusts the scheduling priority in combination with the influence coefficient value, avoiding an unchanging priority mode, and being able to adjust the priority according to the actual task situation. At the scheduling level, it ensures that high-priority tasks enter the queue first, are more likely to obtain resources, ensure the stability of the output time of important tasks, and also releases tedious manual operations, ensuring that important tasks can be completed in time and improving scheduling efficiency.

[0070] Based on the above embodiment, obtaining the dependency relationship between tasks in the task list includes the following steps:

[0071] Step S1021: Obtain a task configuration table for each task in the task list; each task corresponds to a task configuration table.

[0072] In the embodiment of the present application, each task corresponds to a task configuration table to execute the content in the task configuration table.

[0073] If there are too many tasks in a work list, manual operations such as configuring task dependencies in the original Quartz are very inconvenient. Even if it is configured through task definition, when hundreds of tasks need to configure dependencies at the same time, it is a considerable workload and is prone to errors. Later updates and iterations are also inconvenient.

[0074] Therefore, in this step, a TaskSQL script code can be generated for each task configuration table and stored in the metadata database. When the task configuration table is obtained, the task configuration table of each task can be obtained by obtaining its corresponding TaskSQL script code and parsing the TaskSQL script code.

[0075] Step S1022: Read the relationship between each task configuration table through a preset relationship reading tool.

[0076] In one example, the preset relationship reading tool is an automated tool, through which the relationship between each task configuration table can be read. For example, the relationship between each task configuration table is judged by the keywords in each table. Specifically, the SQL (Structured Query Language) task as a whole is calculated according to the calculation process of ODS, DWD, DWS, and ADS. Each task configuration table corresponds to a task, that is, each task will contain SourceTable and TargetTabe. By reading SourceTable and TargetTabe, the relationship between the task configuration tables can be obtained, and then a complete task dependency configuration can be constructed.

[0077] Step S1023: Generate a task dependency configuration table based on the relationship between the task configuration tables.

[0078] In this step, the dependency information parsed in step S1022 is automatically inserted into the task dependency configuration table template to generate a task dependency configuration table.

[0079] Step S1024: Determine the dependency relationship between tasks based on the task dependency configuration table.

[0080] Since one task configuration table corresponds to one task, the dependency relationship between tasks can be determined based on the dependency relationship between task configuration tables. In an example, for example, task A includes subtask B, grandchild task C, and great-grandchild task D, etc., then tasks B, C, and D are downstream tasks of task A, tasks C and D are downstream tasks of task B, and task A is the upstream task of task B. This upstream and downstream relationship is the dependency relationship between tasks. And only after the upstream task is completed can the downstream task be carried out.

[0081] Through the task dependency configuration table, it is convenient to query and analyze the blood relationship information between data. In another example, the task dependency configuration table can also be visualized.

[0082] In a specific embodiment, Figure 2 As shown, determining the dependency relationship between tasks based on the task dependency configuration table includes the following steps:

[0083] Step S201: Refresh the task list.

[0084] Step S202: Determine whether there is a task flow in the task list, if yes, execute step S203, otherwise end.

[0085] Step S203: parse the task flow script code to obtain a task configuration table.

[0086] Step S204: construct the blood relationship data between each task configuration table.

[0087] Step S205: The blood relationship data is stored in a structured manner.

[0088] Step S206: Visually display the blood relationship data.

[0089] The embodiment of the present application obtains dependencies by traversing the task dependency configuration table in the database, which greatly improves the efficiency of obtaining dependencies, reduces configuration errors, and improves the accuracy of obtaining task dependencies.

[0090] In another embodiment, a DAG (Directed Acyclic Graph) of each task may be constructed, and the dependency relationship of each task may be obtained by traversing the DAG.

[0091] Based on the above embodiment, calculating the influence coefficient value of each task on its downstream tasks based on the dependency relationship between the tasks includes the following steps:

[0092] Step S1031: based on the dependency relationship between the tasks, the current number of downstream tasks of each task, the maximum number of downstream tasks among all tasks, and the minimum number of downstream tasks are obtained.

[0093] In this step, the current number of downstream tasks for each task is obtained by traversal, and a new downstream task volume dataset is generated.

[0094] The maximum number of downstream tasks and the minimum number of downstream tasks among all tasks are calculated based on the downstream task volume data set.

[0095] The following is an example of how to calculate the maximum and minimum values ​​in a downstream task dataset:

[0096] The first step is to traverse the downstream task volume dataset: for each data point in the downstream task volume dataset, check its value.

[0097] The second step is to record the maximum and minimum values:

[0098] Initialize two variables, one to store the maximum value (max_value) and the other to store the minimum value (min_alue).

[0099] Traverse each data point in the downstream task volume data set. If the value of the current data point is greater than max_value, update max_value; if the value of the current data point is less than min_value, update min_value.

[0100] Here is a simple Python code example to calculate the maximum and minimum values ​​in a list:

[0101] ```Python

[0102] def find_min_max(data):

[0103] min_value = float('inf') # Initialized to positive infinity

[0104] max_value = float('-inf') # Initialized to negative infinity

[0105] for value in data:

[0106] if value <min_value:

[0107] min_value = value

[0108] if value>max_value:

[0109] max_value = value

[0110] return min_value, max_value

[0111] # Example data

[0112] data = [10, 20, 30, 40, 50]

[0113] min_val, max_val = find_min_max(data)

[0114] print(f"The minimum value is: {min_val}, the maximum value is: {max_val}")

[0115] ```

[0116] Running this code will output: the minimum value is 10 and the maximum value is 50.

[0117] Step S1032: Calculate the influence coefficient value of each task on its downstream tasks based on the current number of downstream tasks, the maximum number of downstream tasks, the minimum number of downstream tasks and preset influence coefficient calculation rules.

[0118] In this step, the maximum number of downstream tasks and the minimum number of downstream tasks can be normalized, and the normalized result is used as the impact coefficient value. The common normalization formula is as follows:

[0119] `normalized_value = (original_value-min_value) / (max_value - min_value)`;

[0120] Among them, original_value represents the current downstream task volume of the target task, min_value represents the minimum downstream task volume, and max_value represents the maximum downstream task volume.

[0121] This normalization formula is used to scale the current downstream task volume of the target task to between 0 and 1, thereby representing the initial impact coefficient value of the target task. Since the initial scheduling priority is in units of 10, as shown in Table 1, the initial impact coefficient value is multiplied by 10 to obtain the final impact coefficient value, which is within 1-10.

[0122] In an example, if the current downstream task volume of the target task is 80, the maximum downstream task volume is 90, and the minimum downstream task volume is 10, then the final impact coefficient value of the target task = [(80-10) / (90-10)]*10=8.75.

[0123] Based on the above embodiment, determining the final scheduling priority based on the initial scheduling priority and the influence coefficient value of each task includes the following steps:

[0124] The initial scheduling priority is adjusted with the value of the influence coefficient as an adjustment parameter to obtain a final scheduling priority.

[0125] In one example, the final scheduling priority can be obtained by adding the setting value in Table 1 to the influence coefficient value. Of course, the final scheduling priority can also be expressed by multiplying the two or other methods that can represent the association between the two, which is not limited here.

[0126] Tasks of the same type consume different types of resources. For example, Spark tasks are memory-intensive, Datax tasks are CPU-intensive, etc. If the same type of tasks are concentrated on one task execution server, it is easy to cause some system resources to be exhausted while other resources are idle. At the same time, if a task execution server has too many concurrent tasks, it is easy to cause memory OOM (Out Of Menmory) and CPU constantly switching process contexts.

[0127] Therefore, based on the above embodiment, a plurality of scheduling queues are provided in the task execution server, each scheduling queue corresponds to a task type and each scheduling queue is provided with a concurrency threshold.

[0128] Allocating tasks to task execution servers based on the final scheduling priority includes:

[0129] Step S1051: Obtain the task type of each task in the task list, the task quantity of each task type, and the remaining resources in the task execution server.

[0130] In one example, the task types include at least import and export tasks for unstructured databases, import and export tasks for relational databases based on DataX, data computing tasks based on Spark / Hive / Flink, real-time collection tasks based on Binlog, remote message sending tasks (supporting EMAIL / WeChat / DingTalk), system control tasks (Shell / Python / Java), etc. These have covered all the task types required by big data. By customizing a variety of task types, the scheduling of most tasks can be met, expanding the practicality of the scheduling system.

[0131] Step S1052: if the remaining amount of resources in the task execution server is greater than the preset resource threshold, then obtaining the concurrent number threshold of each scheduling queue in the task execution server;

[0132] In this step, the task allocation server first detects the remaining resources of each task execution server, such as the storage capacity of memory. A preset resource threshold is set for the memory number in the task executor. If it is less than the resource threshold, the task execution server will no longer participate in the load. The task allocation server can dynamically and evenly distribute tasks to other task execution servers. If it exceeds the preset resource threshold, the task can be executed.

[0133] Step S1053: Determine whether the number of tasks of each task type exceeds the concurrency threshold in the corresponding scheduling queue.

[0134] For example, two scheduling queues are configured in the task execution server, one is a CPU type scheduling queue, and the other is a memory type scheduling queue. The concurrency threshold of each queue is 10, which means that each queue can process 10 tasks of the corresponding type at the same time. In one example, for example, there are 8 spark computing tasks in the task flow, and the task needs to enter the memory type scheduling queue for processing. Since the number of tasks in the task flow is less than 10, there is no need to judge the priority and they can be processed at the same time. However, if there are 12 spark computing tasks in the task flow, since only 10 tasks can be processed at the same time in the memory type scheduling queue, it is necessary to determine which tasks need to be processed first through the following step S1054.

[0135] Step S1054: If exceeded, the task is assigned to the task execution server based on the final scheduling priority.

[0136] The embodiment of the present application sets different scheduling queues according to the task type, allows tasks of the same type to be in the same queue, and sets the maximum concurrency number in each scheduling queue, thereby achieving balanced load on the task execution server, avoiding overloading of some task execution servers while others are idle, and ensuring that the CPU and memory usage of each task execution server remains within a relatively reasonable range, thereby improving the operating efficiency of the scheduling system.

[0137] Since the task execution server is deployed in a distributed manner, partial unavailability will not cause the unavailability of the entire scheduling system. However, the task allocation server has a single point problem. If the task allocation server crashes, all tasks will fail to be dispatched.

[0138] Therefore, based on the above embodiment, the task allocation server includes a main task allocation server and a backup task allocation server, and receiving the task list to be scheduled through the task allocation server includes:

[0139] Monitor the operation status of the main task allocation server in real time through the backup task allocation server; the operation status includes at least normal and fault;

[0140] If the running status is normal, the task list to be scheduled is received through the main task allocation server;

[0141] If the operating status is a failure, the backup task allocation server is switched to receive the task list to be scheduled.

[0142] By adding a backup task allocation server, this application can switch in time when the main task allocation server fails, ensuring the normal progress of the scheduling task, thereby greatly improving the high availability of the scheduling system.

[0143] On the basis of the above embodiments, in order to ensure the stability of the scheduling system, timely alarm notification or even automatic repair when problems occur during scheduling execution, it is necessary to conduct comprehensive monitoring and early warning of the machines, resources, and scheduling tasks of the scheduling cluster. Therefore, the method also includes:

[0144] Step S106: monitor the operation data of the task execution server in real time.

[0145] In the embodiment of the present application, the monitoring can be divided into basic monitoring and log monitoring according to the granularity and dimension:

[0146] Basic monitoring: Combined with operation and maintenance monitoring (process, IO, etc.) and custom monitoring (including task month-on-month fluctuation monitoring, key task output time monitoring, etc.), basic monitoring and alarming of the scheduling server are carried out.

[0147] Log monitoring: By collecting the logs generated when the task is running to Kafka, you can calculate the start and end time of each task, the person in charge, and the amount of resources used (read and write volume, Yarn CPU / memory usage, resource status of the task execution server, etc.).

[0148] You can also analyze the task logs of Hive, MR, Spark, Flink, etc. executed on Hadoop, make a comprehensive diagnosis of GC (Global Catalog), data skew, resource consumption, etc., and finally store the data in NoSQL for further processing and front-end visualization.

[0149] Step S107: determining the task running error type based on the running data, where the task running error type includes a designated error type and a non-designated error type.

[0150] The specified error types refer to some operational error types that are pre-set in the scheduling system, such as temporary error causes such as network jitter, database load, temporary unavailability of service, or timeout caused by busy service.

[0151] Unspecified error types refer to types other than specified error types, such as incorrect upstream table data and script configuration errors.

[0152] Step S108: Perform automatic retry repair based on the specified error type.

[0153] For specified error types, you can set an automatic retry repair strategy to perform retry repair.

[0154] Step S109: Generate remote alarm information based on the non-specified error type.

[0155] In this step, you can configure remote alarms for different non-specified error types and levels, such as DingTalk, WeChat, SMS, and phone calls, to ensure that the operation and maintenance personnel on duty can receive system fault messages in a timely manner 24 hours a day, 7 days a week and take relevant measures for emergency repairs.

[0156] The embodiments of the present application can detect and eliminate errors in the dispatching system in a timely manner through all-round monitoring and remote alarm, thereby ensuring the stability and reliability of the dispatching system.

[0157] Based on the same inventive concept, a big data task scheduling device is provided, such as Figure 3 As shown, the device comprises:

[0158] The receiving unit 301 is used to receive a list of tasks to be scheduled through a task allocation server.

[0159] The task allocation server includes a main task allocation server and a backup task allocation server, and the receiving unit is specifically used for:

[0160] Monitor the operation status of the main task allocation server in real time through the backup task allocation server; the operation status includes at least normal and fault;

[0161] If the running status is normal, the task list to be scheduled is received through the main task allocation server;

[0162] If the operating status is a failure, the backup task allocation server is switched to receive the task list to be scheduled.

[0163] The acquisition unit 302 is used to acquire the initial scheduling priority of each task and the dependency relationship between the tasks in the task list.

[0164] Optionally, the acquisition unit is specifically used for:

[0165] Get the task configuration table of each task in the task list; each task corresponds to a task configuration table;

[0166] Read the relationship between each task configuration table through the preset relationship reading tool;

[0167] Generate a task dependency configuration table based on the relationship between each task configuration table;

[0168] Determine the dependencies between tasks based on the task dependency configuration table.

[0169] The calculation unit 303 is used to calculate the influence coefficient value of each task on its downstream tasks based on the dependency relationship between the tasks.

[0170] The determination unit 304 is configured to determine a final scheduling priority based on the initial scheduling priority and the influence coefficient value of each task.

[0171] The allocating unit 305 is used to allocate each task to a task execution server based on the final scheduling priority.

[0172] In an embodiment of the present application, a plurality of scheduling queues are provided in the task execution server, each scheduling queue corresponds to a task type and each scheduling queue is provided with a concurrency threshold;

[0173] The computing unit is specifically used for:

[0174] Based on the dependencies between tasks, obtain the current number of downstream tasks for each task, the maximum number of downstream tasks among all tasks, and the minimum number of downstream tasks;

[0175] The influence coefficient value of each task on its downstream tasks is calculated based on the current number of downstream tasks, the maximum number of downstream tasks, the minimum number of downstream tasks and the preset influence coefficient calculation rules.

[0176] Determine the unit specifically for:

[0177] The initial scheduling priority and influence coefficient of each task are added together to obtain the final scheduling priority.

[0178] The allocation unit is specifically used for:

[0179] Obtain the task type of each task in the task list, the number of tasks of each task type, and the remaining resources in the task execution server;

[0180] If the remaining resources in the task execution server are greater than the preset resource threshold, the concurrency threshold of each scheduling queue in the task execution server is obtained;

[0181] Determine whether the number of tasks of each task type exceeds the concurrency threshold in the corresponding scheduling queue;

[0182] If it exceeds, the task is assigned to the task execution server based on the final scheduling priority.

[0183] Optionally, the device further comprises a monitoring unit, configured to:

[0184] Real-time monitoring of the operation data of the task execution server;

[0185] Determine a task running error type based on the running data, where the task running error type includes a specified error type and a non-specified error type;

[0186] Automatically retry repair based on specified error types;

[0187] Generates remote warning information based on unspecified error types.

[0188] Based on the same technical concept, an embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a processor 401 , a communication interface 402 , a memory 403 and a communication bus 404 , wherein the processor 401 , the communication interface 402 , and the memory 403 communicate with each other via the communication bus 404 .

[0189] Memory 403, used for storing computer programs;

[0190] The processor 401 is used to implement the steps of the big data task scheduling method when executing the program stored in the memory 403.

[0191] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0192] The communication interface is used for communication between the above electronic device and other devices.

[0193] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0194] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0195] The computer program product for the big data task scheduling method provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0196] The big data task scheduling device provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0197] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0198] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0200] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0201] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0202] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A big data task scheduling method, characterized in that: Applied to the Quartz framework, the method includes: Receiving a list of tasks to be scheduled through a task allocation server; Obtaining the initial scheduling priority of each task and the dependency relationship between the tasks in the task list; the initial scheduling priority is configured based on the priority function supported by Quartz itself; Obtaining the dependency relationship between tasks in the task list includes: Obtain a task configuration table for each task in the task list; each task corresponds to a task configuration table; specifically, generate a TaskSQL script code for each task configuration table, store it in the metadata database, and when obtaining the task configuration table, obtain the corresponding TaskSQL script code, and obtain the task configuration table for each task by parsing the TaskSQL script code; Read the relationship between each task configuration table through the preset relationship reading tool; Generate a task dependency configuration table based on the relationship between the task configuration tables; Determine the dependency relationship between tasks based on the task dependency configuration table; Calculate the influence coefficient of each task on its downstream tasks based on the dependency relationship between the tasks; The steps for calculating the influence coefficient value are as follows: Based on the dependency relationship between the tasks, obtain the current number of downstream tasks for each task, the maximum number of downstream tasks among all tasks, and the minimum number of downstream tasks; Calculate the influence coefficient value of each task on its downstream tasks based on the current number of downstream tasks, the maximum number of downstream tasks, the minimum number of downstream tasks and a preset influence coefficient calculation rule; The preset influence coefficient calculation rule is to calculate the influence coefficient value of each task on its downstream tasks through the normalized formula normalized_value = (original_value-min_value) / (max_value - min_value); Among them, original_value represents the current downstream task volume of the target task, min_value represents the minimum downstream task volume, and max_value represents the maximum downstream task volume; Determine a final scheduling priority based on the initial scheduling priority and the impact coefficient value of each task; Allocating each task to a task execution server based on the final scheduling priority; The task execution server is provided with a plurality of scheduling queues, each of which corresponds to a task type and each of which is provided with a concurrency threshold; the allocating the tasks to the task execution server based on the final scheduling priority comprises: Obtaining the task type of each task in the task list, the number of tasks of each task type, and the remaining resources in the task execution server; If the remaining amount of resources in the task execution server is greater than the preset resource threshold, then obtaining the concurrency threshold of each scheduling queue in the task execution server; Determine whether the number of tasks of each task type exceeds the concurrency threshold in the corresponding scheduling queue; If it exceeds, the task is allocated to the task execution server based on the final scheduling priority.

2. The method according to claim 1, characterized in that The determining of the final scheduling priority based on the initial scheduling priority and the influence coefficient value of each task includes: The initial scheduling priority is adjusted with the value of the influence coefficient as an adjustment parameter to obtain a final scheduling priority.

3. The method according to claim 1, characterized in that The task allocation server includes a main task allocation server and a standby task allocation server, and receiving a list of tasks to be scheduled through the task allocation server includes: Monitor the operating status of the main task allocation server in real time through the backup task allocation server; the operating status includes at least normal and fault; If the operating status is normal, receiving a list of tasks to be scheduled through the main task allocation server; If the operating status is a failure, the backup task allocation server is switched to receive the list of tasks to be scheduled.

4. The method according to claim 1, characterized in that: The method further comprises: Real-time monitoring of the operation data of the task execution server; Determine a task running error type based on the running data, wherein the task running error type includes a specified error type and a non-specified error type; Based on the specified error type, automatically retry repair; Based on the unspecified error type, remote warning information is generated.

5. A big data task scheduling device, characterized in that: Applied to the Quartz framework, the device includes: A receiving unit, used for receiving a list of tasks to be scheduled through a task allocation server; An acquisition unit, used for acquiring an initial scheduling priority of each task and a dependency relationship between the tasks in the task list; the initial scheduling priority is configured based on a priority function supported by Quartz itself; The acquisition unit is specifically used to acquire the task configuration table of each task in the task list; each task corresponds to a task configuration table; specifically, a TaskSQL script code is generated for each task configuration table and stored in the metadata database. When acquiring the task configuration table, the task configuration table of each task is obtained by acquiring the corresponding TaskSQL script code and parsing the TaskSQL script code; Read the relationship between each task configuration table through the preset relationship reading tool; Generate a task dependency configuration table based on the relationship between each task configuration table; Determine the dependencies between tasks based on the task dependency configuration table; A calculation unit, used for calculating the influence coefficient value of each task on its downstream tasks based on the dependency relationship between the tasks; The computing unit is specifically used to obtain the current number of downstream tasks of each task, the maximum number of downstream tasks among all tasks, and the minimum number of downstream tasks based on the dependency relationship between the tasks; Calculate the influence coefficient value of each task on its downstream tasks based on the current number of downstream tasks, the maximum number of downstream tasks, the minimum number of downstream tasks and a preset influence coefficient calculation rule; The preset influence coefficient calculation rule is to calculate the influence coefficient value of each task on its downstream tasks through the normalized formula normalized_value = (original_value-min_value) / (max_value - min_value); Among them, original_value represents the current downstream task volume of the target task, min_value represents the minimum downstream task volume, and max_value represents the maximum downstream task volume; a determining unit, configured to determine a final scheduling priority based on the initial scheduling priority and the influence coefficient value of each task; an allocating unit, configured to allocate each task to a task execution server based on the final scheduling priority; The task execution server is provided with a plurality of scheduling queues, each of which corresponds to a task type and each of which is provided with a concurrent number threshold; the allocation unit is specifically used for: Obtaining the task type of each task in the task list, the number of tasks of each task type, and the remaining resources in the task execution server; If the remaining amount of resources in the task execution server is greater than the preset resource threshold, then obtaining the concurrency threshold of each scheduling queue in the task execution server; Determine whether the number of tasks of each task type exceeds the concurrency threshold in the corresponding scheduling queue; If it exceeds, the task is allocated to the task execution server based on the final scheduling priority.

6. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 4 when executing a program stored in a memory.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.