Task scheduling method, system and equipment based on multi-constraint optimization and storage medium

Abstract

The invention provides a task scheduling method, system and device based on multi-constraint optimization and a storage medium. The method comprises the following steps: acquiring a cluster real-time resource state and historical resource use characteristics of a to-be-executed task; determining constraint information based on the cluster real-time resource state and task configuration metadata of the to-be-executed task; the constraint information comprises an available resource constraint, a task concurrency constraint and a variable domain constraint; substituting the available resource constraint, the task concurrency constraint and the variable domain constraint into a linear programming model to solve the model; and generating a task scheduling strategy according to a model solving result. According to the method, the resource utilization efficiency and the task execution performance in a large-scale distributed computing environment are effectively improved.

Description

Technical Field

[0001] This application belongs to the technical field of task scheduling, and relates to a task scheduling method, system, device and storage medium based on multi-constraint optimization. Background Technology

[0002] The continuous growth of enterprise data scale has become a core feature of the digital age. This trend poses a severe challenge to enterprise data processing capabilities. Traditional centralized architectures are unable to support real-time analysis and elastic needs due to problems such as low processing efficiency, insufficient resource utilization (average only 30%-40%), and difficulty in expansion.

[0003] Distributed computing platforms provide efficient solutions through architectural innovation. Their core advantages include:

[0004] High-efficiency data processing: By adopting distributed computing frameworks (such as Hadoop and Spark) and GPU (Graphics Processing Unit) acceleration technology, the processing speed of massive data can be increased several times to meet real-time requirements.

[0005] Resource optimization and scalability: Virtualization technology enables dynamic allocation and elastic expansion of computing resources. For example, an e-commerce company can temporarily increase its computing power by 300% to cope with traffic peaks, effectively reducing idle waste.

[0006] High availability and disaster recovery capabilities: Distributed architectures (such as multi-datacenter networking) reduce the risk of business interruption and improve system resilience through data redundancy and collaborative management.

[0007] With the continuous growth of enterprise data scale and the widespread application of distributed computing platforms, the scheduling efficiency of big data tasks has become a key factor affecting the overall system performance. Big data task scheduling refers to the orderly arrangement, resource allocation, and dependency management of multiple tasks in the data processing workflow to ensure the efficient and accurate execution of the data processing process. As the data scale and task complexity increase, simple scheduling tools (such as crontab) can no longer meet the requirements. Therefore, it is necessary to introduce a dedicated scheduling system to manage task dependencies, fault tolerance, and resource optimization.

[0008] Traditional task scheduling methods typically employ manual, static orchestration of task dependencies and scheduling times. In this approach, task execution order and start times are often pre-set, lacking awareness of the actual runtime cluster status and the ability to dynamically respond. Because this method cannot flexibly adjust based on real-time resource conditions, it easily leads to uneven distribution of computing resources, with some nodes overloaded while others remain idle, resulting in low overall cluster utilization.

[0009] Furthermore, static scheduling strategies struggle to adapt to the dynamic resource demands of different tasks. Some tasks may experience excessively long waiting times due to resource contention, thus extending the overall task completion cycle. This not only reduces system throughput but also fails to meet the timeliness requirements for task execution in increasingly complex business scenarios. Summary of the Invention

[0010] This application provides a task scheduling method, system, device, and storage medium based on multi-constraint optimization, which is used to solve the problem of improving the overall performance and responsiveness of the system under multiple constraints.

[0011] In a first aspect, this application provides a task scheduling method based on multi-constraint optimization, comprising: acquiring the real-time resource status of a cluster and the historical resource usage characteristics of tasks to be executed; determining constraint information based on the real-time resource status of the cluster and the task configuration metadata of the tasks to be executed; the constraint information including available resource constraints, task concurrency constraints, and variable domain constraints; substituting the available resource constraints, the task concurrency constraints, and the variable domain constraints into a linear programming model and solving the model; and generating a task scheduling strategy based on the model solution results.

[0012] In one implementation of the first aspect, the step of obtaining the real-time resource status of the cluster includes: continuously monitoring and obtaining key resource indicators of each computing node; the key resource indicators include available CPU computing power, remaining memory capacity, disk I / O throughput, and network bandwidth utilization; the relevant data of the key resource indicators are processed by smoothing filtering and persistently stored in time series form.

[0013] In one implementation of the first aspect, the step of obtaining the historical resource usage characteristics of the task to be executed includes: for the task to be executed in the task queue, extracting resource consumption indicators from its historical running records, wherein the resource consumption indicators include peak CPU utilization, average and peak memory usage, I / O operation frequency and data throughput; the resource consumption indicators are aggregated and normalized to construct a standardized profile reflecting the resource demand characteristics of the task.

[0014] In one implementation of the first aspect, the process of establishing the linear programming model includes: defining decision variables based on the tasks to be executed; placing the tasks to be executed in a task queue, with each scheduled task represented by a binary variable, where 1 indicates scheduling and 0 indicates no scheduling; defining parameters based on the real-time resource status of the cluster and the historical resource usage characteristics of the tasks to be executed; the defined parameters include: task parameters, available resource vector, and weight vector; wherein, the task parameters include the set of current tasks and the resource set of the entire system, and the historical resource consumption and corresponding concurrency limit are statistically analyzed for each scheduled task; the objective function is determined based on the defined parameters as follows: ); where N represents a total of N tasks, and j represents each of the N tasks.

[0015] In one implementation of the first aspect, the available resource constraint means that the resources required by the invoked task cannot exceed the current available resources of the cluster; the task concurrency constraint means that the number of scheduled task instances does not exceed its concurrency limit; and the variable domain constraint means that the decision variable must be a binary variable to conform to the discrete characteristics of scheduling decisions.

[0016] In one implementation of the first aspect, the step of generating a task scheduling strategy based on the model solution includes: for scenarios where the task size does not exceed a preset threshold, using a standard mixed-integer programming algorithm to solve the problem, so as to obtain the globally optimal solution within a reasonable time; for ultra-large-scale task queues where the task size exceeds the preset threshold, using a heuristic algorithm to approximate the solution, so as to obtain a near-optimal feasible solution within a finite time; after the standard mixed-integer programming algorithm or any of the heuristic algorithms has been solved, outputting the optimal solution vector as the task scheduling strategy.

[0017] In one implementation of the first aspect, the method further includes: selecting a set of tasks to be scheduled according to the task scheduling strategy, and allocating task instances in the task set to corresponding computing nodes through the cluster API to achieve interaction with the underlying cluster management system.

[0018] Secondly, this application provides a task scheduling system based on multi-constraint optimization, comprising: a resource monitoring module configured to acquire the real-time resource status of the cluster; a task metadata management module configured to statistically analyze the resources required by tasks to determine the historical resource usage characteristics of tasks to be executed; a task queue module configured to place tasks to be scheduled into a task queue; an optimization solution module configured to determine constraint information based on the real-time resource status of the cluster and the task configuration metadata of the tasks to be executed; the constraint information includes available resource constraints, task concurrency constraints, and variable domain constraints; the available resource constraints, task concurrency constraints, and variable domain constraints are substituted into a linear programming model for model solving; a task scheduling strategy is generated based on the model solution results; and a scheduling execution module configured to select a set of tasks to be scheduled according to the task scheduling strategy, and allocate task instances in the task set to corresponding computing nodes through a cluster API, thereby achieving interaction with the underlying cluster management system.

[0019] Thirdly, this application provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method.

[0020] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the method described thereon.

[0021] As described above, the task scheduling method, system, device, and storage medium based on multi-constraint optimization described in this application have the following beneficial effects:

[0022] This application provides a method and system for intelligent scheduling of big data tasks based on multi-constraint optimization. By comprehensively considering the real-time resource status of the cluster, the historical execution characteristics of tasks, and concurrency requirements, the scheduling problem is modeled as a mixed-integer linear programming problem, and solved using operations research optimization theory, thereby achieving global optimization of cluster resource allocation and task scheduling. This invention can effectively improve resource utilization efficiency and task execution performance in large-scale distributed computing environments. Attached Figure Description

[0023] Figure 1 This diagram illustrates the application background of the task scheduling method based on multi-constraint optimization described in the embodiments of this application.

[0024] Figure 2 The diagram shown illustrates the principle flowchart of the task scheduling method based on multi-constraint optimization described in the embodiments of this application.

[0025] Figure 3 The diagram shown is a schematic diagram of the model solution for the task scheduling method based on multi-constraint optimization described in the embodiments of this application.

[0026] Figure 4 The diagram shows the constraint conditions of the task scheduling method based on multi-constraint optimization described in the embodiments of this application.

[0027] Figure 5 The diagram shown is a flowchart of the model solving process for the task scheduling method based on multi-constraint optimization described in the embodiments of this application.

[0028] Figure 6 The diagram shown is a schematic of the solution algorithm for the task scheduling method based on multi-constraint optimization described in the embodiments of this application.

[0029] Figure 7 The diagram shown is an overall schematic of the task scheduling method based on multi-constraint optimization described in the embodiments of this application.

[0030] Figure 8 The diagram shown is a schematic representation of the structural principle of the task scheduling system based on multi-constraint optimization as described in an embodiment of this application.

[0031] Figure 9 The diagram shown is a functional principle diagram of the task scheduling system based on multi-constraint optimization described in the embodiments of this application.

[0032] Figure 10 The diagram shown is a structural connection diagram of the electronic device described in an embodiment of this application.

[0033] Component designation explanation

[0034] 1 electronic devices 11 processor 12 memory 13 Communication interface 14 System bus 8 Task scheduling system based on multi-constraint optimization 81 Resource monitoring module 82 Task metadata management module 83 Task queue module 84 Optimization Solution Module 85 Scheduling Execution Module S21~S24 step S231~S233 step Detailed Implementation

[0035] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0036] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0037] The following embodiments of this application provide a task scheduling method, system, device, and storage medium based on multi-constraint optimization, including but not limited to data computing platforms built on various servers. The following description will take this hardware application scenario as an example.

[0038] Please see Figure 1 The diagram illustrates the application background of the task scheduling method based on multi-constraint optimization described in the embodiments of this application. Figure 1 As shown, this embodiment provides a data computing platform constructed by various servers, specifically including: a web server acting as a task allocator to schedule tasks for server 1, server 2... server n. In practical applications, server 1 can execute a first computing task, server 2 can execute a second computing task, and server n can execute the nth computing task.

[0039] This application dynamically determines the optimal task scheduling sequence based on the real-time resource status of the cluster (such as remaining CPU, memory, and disk I / O) and the historical execution characteristics of tasks (such as resource consumption patterns). By establishing a quantitative optimization model, the utilization rate of cluster resources can be maximized and the total execution time of tasks can be minimized under multiple constraints, thereby significantly improving the overall system performance and responsiveness.

[0040] In this context, a cluster is a group of independent computers (called nodes) connected by a high-speed network to work together to provide high availability, high concurrency processing capabilities, or high-performance computing services.

[0041] Real-time cluster resource status refers to a dynamic snapshot of the actual usage and operating status of various resources (such as CPU, memory, GPU, network, storage, etc.) in the cluster at a specific moment. It reflects the current load level, resource allocation efficiency, and health status of each node, and is a key basis for performance monitoring, troubleshooting, and resource optimization.

[0042] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0043] Please see Figure 2 The diagram shows the principle flowchart of the task scheduling method based on multi-constraint optimization described in the embodiments of this application. Figure 2 As shown, this embodiment provides a task scheduling method based on multi-constraint optimization, which specifically includes the following steps:

[0044] S21, obtain the real-time resource status of the cluster and the historical resource usage characteristics of tasks to be executed.

[0045] In one embodiment, the real-time resource status of the cluster includes the used and remaining amounts of GPU, CPU, memory, and disk I / O. The step of obtaining the real-time resource status of the cluster includes:

[0046] Continuously monitor and acquire key resource indicators for each computing node; these key resource indicators include available CPU computing power, remaining memory capacity, disk I / O throughput, and network bandwidth utilization; the relevant data of these key resource indicators are processed by smoothing and filtering and persistently stored in time series form, which can effectively reduce noise interference.

[0047] In one embodiment, historical resource usage characteristics refer to viewing the average resource consumption during the historical execution of the task, including the usage of GPUs (number), CPUs (number), memory (GB), and disk I / O. The step of obtaining the historical resource usage characteristics of the task to be executed includes:

[0048] For tasks waiting to be executed in the task queue, resource consumption indicators are extracted from their historical running records. These indicators include peak CPU utilization, average and peak memory usage, I / O operation frequency, and data throughput. After aggregation and normalization, these resource consumption indicators are used to construct a standardized profile that reflects the resource requirements of the task.

[0049] Specifically, the tasks to be executed include: SQL (Structured Query Language) scripts, Python scripts, and other program scripts. There is a competition between these scripts, and they need to compete for cluster resources.

[0050] In practical applications, CPU, memory, disk, and network form a standardized structure for a cluster. For example:

[0051] CPU remaining: 200C;

[0052] 10TB of memory remaining;

[0053] Disk read / write speeds are 100MB and 50MB remaining, respectively.

[0054] The network bandwidth is 100M remaining, etc.

[0055] Furthermore, the task concurrency attribute is determined, and the maximum number of concurrent instances is obtained from the task configuration metadata, serving as a key constraint for controlling the scale of concurrent task execution.

[0056] The task concurrency attribute refers to the maximum number of instances or parallelism that a user allows to run a task simultaneously during execution. It determines how many subtasks a task can be broken down into, processed concurrently by multiple execution units (such as threads, processes, or distributed nodes), thus affecting task execution efficiency and resource consumption. Task configuration metadata refers to the data structures and attribute information used to define and configure the tasks required for execution in a specific data processing, analysis, or task orchestration system. The maximum concurrent instance limit refers to the maximum number of instances (such as function instances, task instances, etc.) allowed to be in execution simultaneously in a specific system or service. When the number of concurrent requests increases, the system creates new instances to handle the requests; however, once this limit is reached, new requests will not be able to immediately obtain instance processing but will be queued or directly return a flow control error.

[0057] Furthermore, all collected data, including the real-time cluster resource status, historical resource usage characteristics of tasks to be executed, and task concurrency attributes, are stored in a time-series database (such as Prometheus or InfluxDB), providing reliable data input for subsequent optimization of the scheduling model.

[0058] S22, constraint information is determined based on the real-time resource status of the cluster and the task configuration metadata of the tasks to be executed. The task configuration metadata refers to the historical average resource consumption of each task, including the usage of GPUs (number), CPUs (number), memory (GB), and disk I / O. The constraint information includes available resource constraints, task concurrency constraints, and variable domain constraints. Therefore, this application significantly improves resource utilization by scientifically balancing the differentiated needs of different tasks for various resources (computing, storage, I / O) and modeling task concurrency as a key constraint, achieving "peak shaving" and refined allocation of cluster resources. This approach effectively reduces resource idleness and fragmentation, thereby significantly improving the overall utilization efficiency of key resources such as CPU, memory, and disk I / O.

[0059] S23, Substitute the available resource constraints, the task concurrency constraints, and the variable domain constraints into the linear programming model to solve the model.

[0060] Specifically, this application models a linear programming (MLP) problem, which can be solved in many ways, such as mixed-integer linear programming (MILP), integer programming, etc., to achieve the dual objectives of maximizing system resource utilization and minimizing task execution latency.

[0061] Therefore, this application possesses global resource optimization capabilities. By establishing a mixed-integer linear programming (MILP) model, it coordinates the needs of all available resources and scheduled tasks from a system-wide perspective. This model simultaneously considers multiple resource constraints and task characteristics, avoiding the "local optimum" trap that is easily caused by traditional heuristic rules or local scheduling strategies, and maximizing the overall operational efficiency of the cluster.

[0062] Please see Figure 3 The diagram shows a model solution for the task scheduling method based on multi-constraint optimization described in this application embodiment. Figure 3 As shown, the tasks to be executed in the task queue are optimized using a linear programming model under constraints of available resources, task concurrency, and variable domain to generate a task scheduling strategy.

[0063] Please see Figure 4 The diagram shows the constraint conditions of the task scheduling method based on multi-constraint optimization described in the embodiments of this application. Figure 4 As shown, the specific meanings of available resource constraints, task concurrency constraints, and variable domain constraints are as follows:

[0064] The available resource constraint means that the resources required by the invoked task cannot exceed the current available resources of the cluster.

[0065] The task concurrency constraint means that the number of scheduled task instances does not exceed its concurrency limit.

[0066] The variable domain constraint means that the decision variables must be binary variables, which conforms to the discrete characteristics of scheduling decisions.

[0067] Please see Figure 5 The diagram shows the model solution flowchart of the task scheduling method based on multi-constraint optimization described in the embodiments of this application. Figure 5 As shown, the process of establishing the linear programming model includes:

[0068] S231, Define decision variables according to the tasks to be executed, and place the tasks to be executed in the task queue. Each scheduled task is represented by a binary variable, where 1 indicates scheduling and 0 indicates no scheduling.

[0069] Specifically, scheduled tasks are placed in a queue, and each scheduled task can be represented by a binary variable. , which is the decision variable, where 1 represents scheduling and 0 represents no scheduling.

[0070] S232, Parameters are defined by combining the real-time resource status of the cluster and the historical resource usage characteristics of the tasks to be executed; the defined parameters include: task parameters, available resource vector and weight vector; wherein, the task parameters include the set of current tasks and the resource set of the entire system, and the historical resource consumption and corresponding concurrency limit are statistically analyzed for each scheduled task.

[0071] Specifically, the task parameters are defined as follows:

[0072] Assume the current set of tasks is The entire system's resource set is Where cpu represents the number of CPU cores, mem represents the amount of data in memory, and io represents the maximum disk throughput.

[0073] For each task j, the historical resource consumption is calculated as follows: , This represents the CPU consumption of the j-th task. This represents the memory consumed by the j-th task. This represents the disk I / O consumed by the j-th task.

[0074] The maximum concurrency for each task j is: Once the limit is exceeded, concurrency issues can cause task errors or slow down cluster operation.

[0075] Specifically, the vector of currently available resources in the cluster is defined as follows: .

[0076] Specifically, the weight vector of the resources is defined as follows: This reflects the scarcity and cost of different resources.

[0077] S233, the objective function is determined based on the defined parameters as follows:

[0078] );

[0079] Specifically, we define an objective function to maximize the weighted resource utilization efficiency of the system and maximize the utilization rate of the cluster:

[0080]

[0081] in, This represents the actual concurrency of task j. Here, N represents the total number of tasks, and j represents each of the N tasks.

[0082] Based on the objective function, the specific constraint information is as follows:

[0083] (1) The resources required by the invoked task cannot exceed the current available resources of the cluster. The constraint is set as follows:

[0084]

[0085] (2) Ensure that the number of scheduled task instances does not exceed its concurrency limit. The task concurrency constraint is set as follows:

[0086]

[0087] (3) Ensure that the decision variables are binary variables, which conforms to the discrete characteristics of scheduling decisions. The variable domain constraints are set as follows:

[0088]

[0089] This model is a typical variant of the knapsack problem, belonging to the NP-hard category and requiring specialized optimization algorithms for solution. In practical applications, assuming there are 10,000 tasks to be executed (N=10,000), with 100 tasks running online, but the CPU, memory, and disk are not yet full, leaving space for other tasks to run, how can the remaining tasks be launched to maximize cluster utilization and allow tasks to run faster? However, it must also satisfy the following conditions: each task has a concurrency limit, and the CPU, memory, and disk usage of the entire cluster cannot exceed the cluster's maximum capacity.

[0090] In practical applications, the input information includes: 1. The remaining amount of the cluster's real-time resource status (GPU, CPU, memory, disk I / O); 2. The concurrency of each task to be executed and the data required (GPU, CPU, memory, disk I / O).

[0091] The output information includes: whether each task has started execution (1 for execution, 0 otherwise).

[0092] Example output: Task 1 (x1) = 1, Task 2 (x2) = 0, Task 3 (x3) = 0, Task 4 (x4) = 1. Tasks 1 and 4 start execution, while Tasks 2 and 3 do not execute.

[0093] Therefore, this application possesses good versatility and scalability, and the proposed modeling framework exhibits high flexibility. The resource type set, objective function weights, and constraints in the mathematical model can all be extended and customized according to different business scenarios and cluster architectures (e.g., introducing accelerator resources such as GPUs and FPGAs or network bandwidth constraints), making the scheme applicable to various large-scale distributed computing environments, including but not limited to mainstream platforms such as Hadoop YARN, Kubernetes, or Apache Mesos.

[0094] S24, Generate a task scheduling strategy based on the results of the model solution.

[0095] Please see Figure 6 The diagram shows a schematic representation of the solution algorithm for the task scheduling method based on multi-constraint optimization described in an embodiment of this application. Figure 6 As shown, the steps for generating a task scheduling strategy based on the model solution include:

[0096] (1) For scenarios where the task size does not exceed the preset threshold, the standard mixed integer programming algorithm is used to solve the problem so as to obtain the global optimal solution within a reasonable time.

[0097] Specifically, in this scenario, an exact solution method is adopted. For scenarios with a small task size (N ≤ 1000), a standard mixed integer programming solver (such as CPLEX, Gurobi) is used to solve the problem, which can obtain the global optimal solution in a reasonable time.

[0098] (2) For ultra-large-scale task queues whose task size exceeds the preset threshold, a heuristic algorithm is used to approximate the solution in order to obtain a near-optimal feasible solution within a limited time.

[0099] Specifically, in this scenario, a heuristic algorithm is used. For ultra-large-scale task queues, a heuristic algorithm (such as genetic algorithm, simulated annealing or greedy algorithm) is used to approximate the solution in order to obtain a near-optimal feasible solution in a finite time.

[0100] (3) After the standard mixed integer programming algorithm or any of the heuristic algorithms is solved, the optimal solution vector is output as the task scheduling strategy.

[0101] In one embodiment, the task scheduling method based on multi-constraint optimization further includes:

[0102] The task set to be scheduled is selected according to the task scheduling strategy, and the task instances in the task set are allocated to the corresponding computing nodes through the cluster API to realize interaction with the underlying cluster management system.

[0103] Specifically, after the solution is completed, the optimal solution vector is output. Based on this result, the scheduling system selects the set of tasks to be scheduled and allocates the task instances to the corresponding computing nodes through the cluster API.

[0104] In practical applications, the final solution is... Assume N=11, that is, x N There are a total of 11 tasks, each with a status of 0 or 1. For example, if the result is X = (0,1,0,0,0,0,1,0,0,0,1), it means that the 2nd, 7th, and 11th tasks have started, while the other tasks have not started yet.

[0105] Therefore, this application achieves efficient task execution and throughput improvement, with one of the inherent goals of optimizing the model to shorten the total task completion time. By intelligently selecting the optimal task combination and reducing resource contention and waiting delays between tasks, the completion speed of batch tasks is significantly accelerated, the average job completion time is reduced, and thus the overall throughput of the system is improved.

[0106] Please see Figure 7 The diagram shows the overall task scheduling of the task scheduling method based on multi-constraint optimization described in the embodiments of this application. Figure 7 As shown, the entire process of the task scheduling method based on multi-constraint optimization includes: real-time collection of cluster resources, statistical analysis of the resources required by tasks, placing the tasks to be scheduled into a task queue, and then establishing an intelligent optimization model, namely a linear programming model. The task scheduling strategy is obtained by solving the linear programming model, and then tasks are executed according to the task scheduling strategy.

[0107] In practical applications, taking big data platforms as an example, their workloads typically include two core tasks: SQL-based data processing pipelines (such as data ingestion, integration, and logical processing) and Python-based AI jobs. Currently, these tasks generally rely on manually orchestrated scheduling strategies, which can easily lead to resource allocation imbalances. During peak business periods, the number of high-concurrency tasks may exceed the cluster's processing capacity limit, resulting in task queue congestion and various concurrency conflicts; while during off-peak periods, computing resources are idle due to a lack of tasks, leading to low overall resource utilization. This application, through an intelligent scheduling mechanism, can dynamically coordinate different types of tasks, effectively avoiding concurrency bottlenecks while maximizing cluster resource utilization.

[0108] The data processing pipeline scheduling process includes: the scheduling system automatically starts at midnight every day, first executing the data ingestion task in parallel, synchronizing data from various sources to the data lake storage; after all ingestion tasks are successful, the data integration task is triggered to clean, correlate, and standardize the data, generating a detailed data layer; then, the data processing tasks for each theme are started in parallel to aggregate and summarize the detailed data, forming a data mart layer; finally, after all processing tasks are completed, the indicator calculation task is started to generate core business indicators, which are automatically output to the BI (Business Intelligence) system, cache, and reporting services. The entire process ensures that the transformation from raw data to decision support data is completed efficiently and reliably within the specified time through strict dependency control and multi-layer quality checks.

[0109] For AI jobs based on Python scripts, once the above data processing pipeline tasks are completed, it will automatically trigger the execution of analysis such as model prediction, anomaly detection, or intelligent recommendation, store the AI ​​results in a dedicated database and push them to business applications, forming an end-to-end automated pipeline from raw data to intelligent decision-making.

[0110] Therefore, this application solves the problems of low cluster resource utilization and low task execution efficiency caused by static task orchestration in the prior art. This invention transforms the task scheduling problem into a resource allocation problem in operations research by establishing a precise mathematical optimization model, and uses mathematical programming methods to achieve global optimization of system operating efficiency.

[0111] The scope of protection of the task scheduling method based on multi-constraint optimization described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0112] This application also provides a task scheduling system based on multi-constraint optimization. The task scheduling system based on multi-constraint optimization can implement the task scheduling method based on multi-constraint optimization described in this application. However, the implementation device of the task scheduling method based on multi-constraint optimization described in this application includes, but is not limited to, the structure of the task scheduling system based on multi-constraint optimization listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0113] Please see Figure 8 The diagram shows the structural principle of the task scheduling system based on multi-constraint optimization described in the embodiments of this application. Figure 8 As shown, this embodiment provides a task scheduling system 8 based on multi-constraint optimization, which specifically includes: a resource monitoring module 81, a task metadata management module 82, a task queue module 83, an optimization solution module 84, and a scheduling execution module 85.

[0114] The resource monitoring module 81 is configured to obtain the real-time resource status of the cluster.

[0115] In one embodiment, the resource monitoring module 81 is further configured to continuously monitor and acquire key resource indicators of each computing node; the key resource indicators include available CPU computing power, remaining memory capacity, disk I / O throughput, and network bandwidth utilization; the relevant data of the key resource indicators are processed by smoothing and filtering and persistently stored in time series form.

[0116] The task metadata management module 82 is configured to count the resources required by a task in order to determine the historical resource usage characteristics of the task to be executed.

[0117] In one embodiment, the task metadata management module 82 is further configured to extract resource consumption indicators from the historical running records of tasks to be executed in the task queue. The resource consumption indicators include peak CPU utilization, average and peak memory usage, I / O operation frequency and data throughput. The resource consumption indicators are aggregated and normalized to construct a standardized profile that reflects the resource demand characteristics of the task.

[0118] The task queue module 83 is configured to place the tasks to be scheduled into the task queue.

[0119] The optimization solution module 84 is configured to determine constraint information based on the real-time resource status of the cluster and the task configuration metadata of the task to be executed; the constraint information includes available resource constraints, task concurrency constraints, and variable domain constraints; the available resource constraints, the task concurrency constraints, and the variable domain constraints are substituted into the linear programming model to solve the model; and a task scheduling strategy is generated based on the model solution results.

[0120] In one embodiment, the available resource constraint means that the resources required by the invoked task cannot exceed the current available resources of the cluster; the task concurrency constraint means that the number of scheduled task instances does not exceed its concurrency limit; and the variable domain constraint means that the decision variable must be a binary variable to conform to the discrete characteristics of scheduling decisions.

[0121] In one embodiment, the process of establishing the linear programming model includes:

[0122] Decision variables are defined based on the tasks to be executed, and the tasks to be executed are placed in the task queue. Each scheduled task is represented by a binary variable, where 1 indicates scheduling and 0 indicates no scheduling.

[0123] The parameters are defined by combining the real-time resource status of the cluster and the historical resource usage characteristics of the tasks to be executed; the defined parameters include: task parameters, available resource vector, and weight vector; wherein, the task parameters include the set of current tasks and the resource set of the entire system, and the historical resource consumption and corresponding concurrency limit are statistically analyzed for each scheduled task.

[0124] The objective function is determined based on the defined parameters as follows:

[0125] );

[0126] Where N represents a total of N tasks, and j represents each of the N tasks.

[0127] In one embodiment, the optimization solution module 84 is further configured to use a standard mixed integer programming algorithm to solve the problem for scenarios where the task size does not exceed a preset threshold, so as to obtain the global optimal solution within a reasonable time; and to use a heuristic algorithm to approximate the solution for ultra-large-scale task queues where the task size exceeds the preset threshold, so as to obtain a near-optimal feasible solution within a finite time. After the standard mixed integer programming algorithm or any of the heuristic algorithms is completed, the optimal solution vector is output as the task scheduling strategy.

[0128] The scheduling execution module 85 is configured to select a set of tasks to be scheduled according to the task scheduling strategy, and allocate task instances in the task set to corresponding computing nodes through the cluster API, thereby realizing interaction with the underlying cluster management system.

[0129] Please see Figure 9 The diagram shows the functional principle of the task scheduling system based on multi-constraint optimization described in the embodiments of this application. Figure 9 As shown, the entire process of the task scheduling system based on multi-constraint optimization includes: the resource monitoring module 81 collects cluster resources in real time; the task metadata management module 82 counts the resources required by the task; the task queue module 83 places the tasks to be scheduled into the task queue; the optimization solution module 84 establishes an intelligent optimization model, i.e., a linear programming model, and obtains the task scheduling strategy by solving the linear programming model; and then the scheduling execution module 85 runs the task according to the task scheduling strategy.

[0130] Therefore, this application provides a task intelligent scheduling method for large-scale computing clusters, which can dynamically and intelligently decide the task scheduling order and resource allocation strategy based on the real-time resource status of the cluster and the historical execution characteristics of the tasks. This solves the problems caused by the traditional reliance on manual static arrangement of task logic and scheduling time, thereby significantly improving the cluster resource utilization and effectively reducing the overall task running time. It realizes closed-loop automation from data perception to optimization decision-making to scheduling execution, and improves the intelligence level and operating efficiency of large-scale data clusters.

[0131] In the embodiments provided in this application, it should be understood that the disclosed systems or methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or modules or units, and may be electrical, mechanical, or other forms.

[0132] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0133] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] This application provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method described thereon.

[0135] Please see Figure 10 The diagram shows the structural connections of the electronic device described in the embodiments of this application. Figure 10 As shown, the electronic device 1 of this application includes: a processor 11, a memory 12, a communication interface 13 and / or a system bus 14. The memory 12 and the communication interface 13 are connected to the processor 11 through the system bus 14 and complete communication between them. The memory 12 is used to store computer programs, the communication interface 13 is used to communicate with other devices, and the processor 11 is used to run the computer programs, causing the electronic device 1 to execute the various steps of the task scheduling method based on multi-constraint optimization.

[0136] The processor 11 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0137] The aforementioned memory 12 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0138] The system bus 14 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus 14 can be divided into an address bus, a data bus, a control bus, etc. The communication interface is used to enable communication between the database access device and other devices (such as clients, read-write databases, and read-only databases).

[0139] The task scheduling method based on multi-constraint optimization includes: acquiring the real-time resource status of the cluster and the historical resource usage characteristics of the tasks to be executed; determining constraint information based on the real-time resource status of the cluster and the task configuration metadata of the tasks to be executed; the constraint information includes available resource constraints, task concurrency constraints, and variable domain constraints; substituting the available resource constraints, the task concurrency constraints, and the variable domain constraints into a linear programming model and solving the model; and generating a task scheduling strategy based on the model solution results.

[0140] In one embodiment, the step of obtaining the real-time resource status of the cluster includes: continuously monitoring and obtaining key resource indicators of each computing node; the key resource indicators include available CPU computing power, remaining memory capacity, disk I / O throughput, and network bandwidth utilization; the relevant data of the key resource indicators are processed by smoothing and filtering and persistently stored in time series form.

[0141] In one embodiment, the step of obtaining the historical resource usage characteristics of a task to be executed includes: for a task to be executed in a task queue, extracting resource consumption indicators from its historical running records, wherein the resource consumption indicators include peak CPU utilization, average and peak memory usage, I / O operation frequency and data throughput; the resource consumption indicators are aggregated and normalized to construct a standardized profile reflecting the resource demand characteristics of the task.

[0142] In one embodiment, the process of establishing the linear programming model includes: defining decision variables based on the tasks to be executed; placing the tasks to be executed in a task queue, with each scheduled task represented by a binary variable, where 1 indicates scheduling and 0 indicates no scheduling; defining parameters based on the real-time resource status of the cluster and the historical resource usage characteristics of the tasks to be executed; the defined parameters include: task parameters, available resource vector, and weight vector; wherein, the task parameters include the set of current tasks and the resource set of the entire system, and the historical resource consumption and corresponding concurrency limit are statistically analyzed for each scheduled task; and the objective function is determined based on the defined parameters as follows: ); where N represents a total of N tasks, and j represents each of the N tasks.

[0143] In one embodiment, the available resource constraint means that the resources required by the invoked task cannot exceed the current available resources of the cluster; the task concurrency constraint means that the number of scheduled task instances does not exceed its concurrency limit; and the variable domain constraint means that the decision variable must be a binary variable to conform to the discrete characteristics of scheduling decisions.

[0144] In one embodiment, the step of generating a task scheduling strategy based on the model solution includes: for scenarios where the task size does not exceed a preset threshold, using a standard mixed-integer programming algorithm to solve the problem, so as to obtain the globally optimal solution within a reasonable time; for ultra-large-scale task queues where the task size exceeds the preset threshold, using a heuristic algorithm to approximate the solution, so as to obtain a near-optimal feasible solution within a finite time; after the standard mixed-integer programming algorithm or any of the heuristic algorithms has been solved, outputting the optimal solution vector as the task scheduling strategy.

[0145] In one embodiment, the method further includes: selecting a set of tasks to be scheduled according to the task scheduling strategy, and allocating task instances in the task set to corresponding computing nodes through the cluster API to achieve interaction with the underlying cluster management system.

[0146] In practical applications, the electronic device may be a computer including all or some of its components such as memory, storage controller, one or more processing units (CPU), peripheral interfaces, RF circuits, audio circuits, speakers, microphones, input / output (I / O) subsystems, displays, other output or control devices, and external ports; the computer includes, but is not limited to, personal computers such as desktop computers, laptops, tablets, smartphones, smart TVs, and personal digital assistants (PDAs). In other embodiments, the electronic device may also be a server, which may be deployed on one or more physical servers depending on factors such as function and load, or it may be a cloud server composed of distributed or centralized server clusters; this embodiment does not limit this.

[0147] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed, implements the task scheduling method based on multi-constraint optimization.

[0148] The task scheduling method based on multi-constraint optimization includes: acquiring the real-time resource status of the cluster and the historical resource usage characteristics of the tasks to be executed; determining constraint information based on the real-time resource status of the cluster and the task configuration metadata of the tasks to be executed; the constraint information includes available resource constraints, task concurrency constraints, and variable domain constraints; substituting the available resource constraints, the task concurrency constraints, and the variable domain constraints into a linear programming model and solving the model; and generating a task scheduling strategy based on the model solution results.

[0149] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0150] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0151] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A task scheduling method based on multi-constraint optimization, characterized in that, include: Obtain the real-time resource status of the cluster and the historical resource usage characteristics of tasks to be executed; Constraint information is determined based on the real-time resource status of the cluster and the task configuration metadata of the task to be executed; the constraint information includes available resource constraints, task concurrency constraints, and variable domain constraints; Substitute the available resource constraints, the task concurrency constraints, and the variable domain constraints into the linear programming model to solve the model; A task scheduling strategy is generated based on the results of the model solution.

2. The method according to claim 1, characterized in that, The steps to obtain the real-time resource status of the cluster include: Continuously monitor and acquire key resource indicators for each computing node; these key resource indicators include available CPU computing power, remaining memory capacity, disk I / O throughput, and network bandwidth utilization; the relevant data of these key resource indicators are processed by smoothing and filtering and persistently stored in time series format.

3. The method according to claim 1, characterized in that, The steps for obtaining historical resource usage characteristics of a task to be executed include: For tasks waiting to be executed in the task queue, resource consumption indicators are extracted from their historical running records. These indicators include peak CPU utilization, average and peak memory usage, I / O operation frequency, and data throughput. After aggregation and normalization, these resource consumption indicators are used to construct a standardized profile that reflects the resource requirements of the task.

4. The method according to claim 1, characterized in that, The process of establishing the linear programming model includes: Decision variables are defined based on the tasks to be executed, and the tasks to be executed are placed in the task queue. Each scheduled task is represented by a binary variable, where 1 indicates scheduling and 0 indicates no scheduling. The parameters are defined by combining the real-time resource status of the cluster and the historical resource usage characteristics of the tasks to be executed; the defined parameters include: task parameters, available resource vector, and weight vector; wherein, the task parameters include the set of current tasks and the resource set of the entire system, and the historical resource consumption and corresponding concurrency limit are statistically analyzed for each scheduled task. The objective function is determined based on the defined parameters as follows: ）； Where N represents a total of N tasks, and j represents each of the N tasks.

5. The method according to claim 1, characterized in that: The available resource constraint means that the resources required by the invoked task cannot exceed the current available resources of the cluster. The task concurrency constraint means that the number of scheduled task instances does not exceed its concurrency limit; The variable domain constraint means that the decision variables must be binary variables, which conforms to the discrete characteristics of scheduling decisions.

6. The method according to claim 1, characterized in that, The steps for generating a task scheduling strategy based on the model solution include: For scenarios where the task size does not exceed a preset threshold, a standard mixed-integer programming algorithm is used to solve the problem in order to obtain the global optimal solution within a reasonable time. For ultra-large-scale task queues whose task size exceeds the preset threshold, a heuristic algorithm is used to approximate the solution in order to obtain a near-optimal feasible solution within a limited time. After the standard mixed integer programming algorithm or any of the heuristic algorithms is completed, the optimal solution vector is output as the task scheduling strategy.

7. The method according to claim 1, characterized in that, Also includes: The task set to be scheduled is selected according to the task scheduling strategy, and the task instances in the task set are allocated to the corresponding computing nodes through the cluster API to realize interaction with the underlying cluster management system.

8. A task scheduling system based on multi-constraint optimization, characterized in that, include: The resource monitoring module is configured to obtain the real-time resource status of the cluster; The task metadata management module is configured to collect statistics on the resources required by a task in order to determine the historical resource usage characteristics of the task to be executed. The task queue module is configured to place scheduled tasks into the task queue; The optimization solution module is configured to determine constraint information based on the real-time resource status of the cluster and the task configuration metadata of the task to be executed; the constraint information includes available resource constraints, task concurrency constraints, and variable domain constraints; the available resource constraints, task concurrency constraints, and variable domain constraints are substituted into a linear programming model to solve the model; and a task scheduling strategy is generated based on the model solution results. The scheduling and execution module is configured to select a set of tasks to be scheduled according to the task scheduling strategy, and allocate task instances in the task set to the corresponding computing nodes through the cluster API, thereby realizing interaction with the underlying cluster management system.

9. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the method described in any one of claims 1 to 7.

Images