Intelligent scheduling method, system, device and medium for big data task

Abstract

The application provides a kind of big data task intelligent scheduling method, system, equipment and medium, the method is applied to big data task intelligent scheduling system, the big data task intelligent scheduling system includes real-time monitoring module, resource demand evaluation module, dynamic resource allocation module, exception handling module and data storage module.The application builds real-time monitoring, accurate evaluation, dynamic allocation and intelligent fault-tolerant full-process automation closed-loop system, accurately solves the core problems such as resource monitoring lag, low allocation accuracy, dependence on manual scheduling and weak pertinence of exception handling, realizes the technical effects of resource utilization rate improvement, task execution efficiency optimization, operation and maintenance cost reduction and system stability enhancement.

Description

Technical Field

[0001] This invention relates to the field of AI accelerator technology, and in particular to an intelligent scheduling method, system, device and medium for big data tasks. Background Technology

[0002] With the development of the digital economy and the deepening application of big data / AI, large-scale computing tasks place higher demands on the scheduling and management of underlying resources. However, existing resource management solutions (such as Apache YARN) mainly rely on static allocation frameworks, that is, resources are requested on demand when a task starts and run until completion, lacking dynamic adaptation capabilities. This model exposes obvious limitations in the current environment:

[0003] Insufficient real-time monitoring: Existing monitoring systems often use periodic sampling intervals of tens of seconds or even minutes, making it difficult to promptly capture sudden changes in resource demand or hardware failures, easily leading to task lag or even crashes. Low resource utilization: Allocation strategies are primarily based on user submissions, failing to consider the dynamic characteristics of tasks and the real-time dynamics of the resource pool, resulting in idle and wasted resources, with an average cluster utilization rate of only 35%-50%. Lack of dynamic resource allocation mechanism: Resource scheduling relies on manual intervention, resulting in slow response, low efficiency, and difficulty in adapting to the needs of large-scale automated tasks, leading to high operation and maintenance costs. Inadequate exception handling mechanism: For task failures or malfunctions, simple retry or termination strategies are typically used, lacking the ability to automatically locate faults and resume execution, resulting in poor task execution reliability. In summary, the deficiencies of existing solutions in real-time monitoring, allocation accuracy, dynamic adaptation, and fault tolerance directly lead to wasted computing power, unstable task operation, and excessively high operation and maintenance costs, failing to meet the needs of large-scale dynamic big data tasks. Therefore, an effective intelligent scheduling method for big data tasks is urgently needed to solve the above problems. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide an intelligent scheduling method, system, device and medium for big data tasks that overcomes or at least partially solves the above problems.

[0005] To achieve the above and other related objectives, this invention provides an intelligent scheduling method for big data tasks, applied to an intelligent scheduling system for big data tasks. The intelligent scheduling system for big data tasks includes a real-time monitoring module, a resource demand assessment module, a dynamic resource allocation module, an exception handling module, and a data storage module. The method includes: The real-time monitoring module collects key indicators of containerized task execution and the hardware resource status of cluster nodes in real time, and generates structured real-time monitoring data after preprocessing. By integrating historical and real-time monitoring data through the resource demand assessment module, and combining the differences in resource supply and demand with the characteristics of the task execution stage, multi-dimensional correlation analysis is performed to output resource configuration suggestions and allocation priority strategies for the entire task lifecycle. The dynamic resource allocation module, based on the improved fair scheduling algorithm and logical resource partitioning mechanism, combined with the resource configuration suggestions, the allocation priority strategy and the real-time load status of each logical resource sub-pool, formulates the optimal scheduling strategy, and realizes accurate allocation of resources and tasks through the cluster management interface, and dynamically reclaims and redistributes idle resources throughout the entire life cycle of the task. When a task execution failure is detected, an anomaly root cause analysis mechanism that links resource usage status with task execution results is initiated through the anomaly handling module. Combined with the real-time monitoring data, a linked diagnosis is carried out to verify time synchronization and demand adaptability. If it is determined that the task failure is caused by a resource allocation problem, an adaptive response mechanism is triggered. The resource configuration of the task is adjusted through the dynamic resource allocation module, the optimal scheduling strategy is updated, and the task instance is restarted to achieve task recovery.

[0006] Optionally, the real-time monitoring module includes a task status acquisition unit, a resource status acquisition unit, and a data preprocessing unit. The real-time monitoring module acquires key indicators of containerized task execution and the hardware resource status of cluster nodes in real time, and generates structured real-time monitoring data through preprocessing, including: Key metrics for containerized task execution are collected in real time through the task status acquisition unit. The hardware resource status of cluster nodes is collected in real time through the resource status acquisition unit; The data preprocessing unit cleans, standardizes, and extracts features from the key indicators and hardware resource status to output structured real-time monitoring data.

[0007] Optionally, the resource demand assessment module includes a data integration unit and a resource demand suggestion unit. The resource demand assessment module integrates historical monitoring data and real-time monitoring data, and performs multi-dimensional correlation analysis based on resource supply and demand differences and task execution stage characteristics to output resource allocation suggestions and allocation priority strategies for the entire task lifecycle, including: The data integration unit acquires historical monitoring data and real-time monitoring data, and categorizes and integrates them according to data dimensions to generate a unified integrated monitoring dataset. By combining the resource demand suggestion unit with the characteristics of the task execution stage and the differences between resource supply and demand, a multi-dimensional correlation analysis is performed on the integrated monitoring dataset to output resource configuration suggestions and allocation priority strategies for each stage of the task's entire life cycle.

[0008] Optionally, the dynamic resource allocation module includes a resource scheduling engine and a resource pool management unit. The dynamic resource allocation module, based on an improved fair scheduling algorithm and logical resource partitioning mechanism, and combining the resource configuration suggestions, the allocation priority strategy, and the real-time load status of each logical resource sub-pool, formulates an optimal scheduling strategy, including: The resource pool management unit collects the raw load data of each logical resource sub-pool in real time according to the logical resource partitioning mechanism. After preprocessing by filtering instantaneous peak values ​​and removing invalid data, the real-time load status of each logical resource sub-pool is generated. The resource scheduling engine performs standardized processing on the resource configuration suggestions, allocation priority strategies, and real-time load status of each logical resource sub-pool, including unified resource measurement units, completion of missing fields, and removal of outliers, to generate a standardized dataset. The resource scheduling engine, based on the improved fair scheduling algorithm and the standardized dataset, sequentially completes task and logical resource sub-pool adaptation, priority sorting and fairness calibration, resource supply and demand balance calculation, and finally dynamically formulates the optimal scheduling strategy.

[0009] Optionally, the step of accurately allocating resources and tasks through a cluster management interface and dynamically reclaiming and reallocating idle resources throughout the task's lifecycle includes: Based on the optimal scheduling strategy, the resource scheduling engine performs container creation, resource binding, and task scheduling operations through the cluster management interface, accurately matching target resources and tasks in the logical resource sub-pool and completing on-demand resource allocation. Based on the resource allocation execution results of the dynamic resource allocation module, the resource pool management unit updates the load status of each logical resource sub-pool in real time and synchronizes it to the resource scheduling engine. The resource scheduling engine monitors the usage status of allocated resources in real time throughout the entire task lifecycle and triggers a recycling mechanism for resources that have reached the idle threshold, returning the recycled idle resources to the corresponding logical resource sub-pool. The resource scheduling engine dynamically adjusts the subsequent optimal scheduling strategy based on the updated logical resource sub-pool load status synchronized by the resource pool management unit and the improved fair scheduling algorithm. It also prioritizes the reallocation of reclaimed idle resources to tasks with urgent resource needs, combining resource supply and demand difference data with allocation priority strategies, thereby achieving dynamic recycling of resources.

[0010] Optionally, the exception handling module includes an exception detection unit and an optimization and restart unit. When a task execution failure is detected, the exception handling module initiates an exception root cause analysis mechanism that links resource usage status with task execution results. This mechanism, combined with real-time monitoring data, performs linked diagnostics to verify time synchronization and requirement adaptability, including: The system responds to task execution failure events through an anomaly detection unit, receives real-time monitoring data via a high-speed data bus, and extracts resource usage status data and task execution status data after data parsing. The resource usage status data includes the task's CPU utilization, memory utilization, resource allocation, and resource load fluctuations. The task execution status data includes the task failure flag, failure timestamp, and failure log details. The anomaly detection unit performs a linkage analysis on the resource usage status data and the task execution status data, and performs time synchronization and demand adaptability verification in multiple dimensions. Specifically, it determines the time overlap between the task failure time and the resource anomaly status to verify whether the resource anomaly occurred before or simultaneously with the task failure event, and clarifies the temporal correlation between the two. In addition, it compares the current resource allocation amount with the resource configuration suggestions output by the resource demand assessment module to verify whether the resource allocation is lower than the task demand lower limit, whether there is a resource allocation imbalance, or whether there is insufficient available resources due to resource competition.

[0011] Optionally, if the task failure is determined to be caused by a resource allocation problem, an adaptive response mechanism is triggered. This mechanism adjusts the task's resource configuration, updates the optimal scheduling strategy, and restarts the task instance through the dynamic resource allocation module to achieve task recovery. This includes: After the anomaly detection unit determines that the task failure was caused by a resource allocation problem, it outputs the determination result of the failure caused by the resource allocation problem and the anomaly details to the optimization and restart unit. The optimization and restart unit sends a two-way request to the dynamic resource allocation module to obtain the historical resource allocation record of the task and submit targeted resource configuration adjustment requirements. By combining the historical resource allocation records of the task, the real-time load status of the current logical resource sub-pool, and the resource configuration suggestions, the dynamic resource allocation module completes the precise adjustment of the resource configuration of the task and updates the optimal scheduling strategy simultaneously. Then, the resource adjustment results are fed back to the optimization and restart unit. After receiving the resource adjustment result through the optimization and restart unit and confirming that the adjusted resource configuration meets the task requirements and does not exceed the resource limit of the current logical resource sub-pool, the unit sends a restart command to the task execution container through the cluster management interface, so that the task can be re-executed in a sufficient and suitable resource environment.

[0012] Secondly, the present invention also provides an intelligent scheduling system for big data tasks, the system comprising: The real-time monitoring module is used to collect key indicators of containerized task execution and the hardware resource status of cluster nodes in real time, and generate structured real-time monitoring data after preprocessing. The resource demand assessment module is used to integrate historical and real-time monitoring data, and combine the differences in resource supply and demand with the characteristics of the task execution stage to conduct multi-dimensional correlation analysis, and output resource configuration suggestions and allocation priority strategies for the entire life cycle of the task. The dynamic resource allocation module is used to formulate the optimal scheduling strategy based on the improved fair scheduling algorithm and logical resource partitioning mechanism, combined with the resource configuration suggestions, the allocation priority strategy and the real-time load status of each logical resource sub-pool, and to realize the precise allocation of resources and tasks through the cluster management interface, and to dynamically reclaim and redistribute idle resources throughout the entire life cycle of the task. The exception handling module is used to initiate an exception root cause analysis mechanism that links resource usage status with task execution results when a task execution failure is detected. It combines the real-time monitoring data to carry out linked diagnosis to verify time synchronization and demand adaptability. If it is determined that the task failure is caused by a resource allocation problem, an adaptive response mechanism is triggered. The dynamic resource allocation module adjusts the resource configuration of the task, updates the optimal scheduling strategy, and restarts the task instance to achieve task recovery.

[0013] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to enable the electronic device to perform the steps of the intelligent scheduling method for big data tasks as described above.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the steps of the intelligent scheduling method for big data tasks as described above.

[0015] Fifthly, the present invention provides a computer program product including computer program code, which, when run on a computer, causes the computer to implement the method described above.

[0016] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects: This invention precisely solves core problems such as lagging resource monitoring, low allocation accuracy, reliance on manual scheduling, and weak targeted anomaly handling by constructing a fully automated closed-loop system that integrates real-time monitoring, accurate evaluation, dynamic allocation, and intelligent fault tolerance. This results in improved resource utilization, optimized task execution efficiency, reduced operation and maintenance costs, and enhanced system stability. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart of an intelligent scheduling method for big data tasks in one embodiment of the present invention.

[0018] Figure 2 The diagram shows a functional module schematic of an intelligent scheduling system for big data tasks in one embodiment of the present invention.

[0019] Figure 3 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0022] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0023] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0024] Unless otherwise stated, the term "multiple" means two or more.

[0025] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0026] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0027] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0028] Please see Figure 1 An embodiment of the present invention provides an intelligent scheduling method for big data tasks, applied to an intelligent scheduling system for big data tasks. The intelligent scheduling system for big data tasks includes a real-time monitoring module, a resource demand assessment module, a dynamic resource allocation module, an exception handling module, and a data storage module. The method may include the following steps S10~S40:

[0029] Step S10: The key indicators of containerized task execution and the hardware resource status of cluster nodes are collected in real time through the real-time monitoring module, and structured real-time monitoring data is generated after preprocessing.

[0030] Key metrics, which are core indicators used to characterize the running status, resource consumption, and execution effect of containerized tasks, include, but are not limited to: task running progress, real-time CPU / memory utilization, task execution latency, abnormal alarm information, and task stage identifiers (such as startup stage, computation stage, IO stage, and idle stage).

[0031] Hardware resource status refers to the core parameters used to characterize the hardware operating status and availability of cluster nodes, including but not limited to: total CPU load, remaining memory capacity, disk I / O throughput, network bandwidth utilization, and remaining storage space.

[0032] Real-time monitoring data is structured data used to characterize the real-time status of cluster hardware resources, the execution status of containerized tasks, and the allocation and usage of resources. It includes, but is not limited to: real-time cluster resource status, task resource application information, and real-time usage data of allocated resources.

[0033] In practical implementation, JProfiler technology, deployed within the execution container of each big data computing task in the real-time monitoring module, can collect key metrics of containerized task execution in real time, and collect hardware resource status through kernel-mode probes deployed on each node of the cluster. Then, the Flink streaming computing framework is used to preprocess the collected key metrics and hardware resource status (such as data cleaning, outlier removal, and format standardization) to generate structured real-time monitoring data. Subsequently, the real-time monitoring data can be synchronized to the resource requirement assessment module and the data storage module respectively, providing standardized data input for the downstream resource requirement assessment module.

[0034] Step S20: By integrating historical monitoring data and real-time monitoring data through the resource demand assessment module, and combining the differences in resource supply and demand with the characteristics of the task execution stage, multi-dimensional correlation analysis is performed to output resource configuration suggestions and allocation priority strategies for the entire task lifecycle.

[0035] Among them, historical monitoring data is used to characterize the historical structured data collected by the real-time monitoring module and stored in the data storage module after preprocessing. It characterizes the status of cluster hardware resources, the execution status of containerized tasks, and the allocation and use of resources. It includes, but is not limited to, historical cluster resource load data, historical task resource consumption data, historical task execution effect data, and historical abnormal log data, providing historical data support for the multi-dimensional correlation analysis of the resource demand assessment module.

[0036] Resource supply and demand discrepancy is a core analytical dimension used to characterize the quantitative deviation between the current hardware resource supply capacity of the cluster and the resource demand at each stage of the containerized task's lifecycle. Specifically, it manifests as a mismatch between supply and demand. Its core is to use historical and real-time monitoring data, combined with task execution stage characteristics (startup / computation / IO / idle stages) and task attributes, to quantitatively characterize the static absolute difference, relative matching degree, and dynamic trends between the cluster's resource supply capacity and task resource demand from multiple dimensions. This accurately identifies supply and demand mismatch patterns under different resource types (CPU, memory, GPU, HBM bandwidth, IO bandwidth, etc.), different task stages, different time dimensions, and different task types, providing a quantitative decision-making basis for the resource demand assessment module to formulate accurate resource allocation recommendations and allocation priority strategies.

[0037] Resource configuration recommendations are quantitative parameter schemes used to characterize the optimal resource requirements of containerized tasks at each execution stage of the entire lifecycle (such as startup, computation, IO, and idle stages), including but not limited to specific allocatable parameters such as the number of CPU cores, memory capacity, network bandwidth, and disk IO quota.

[0038] Priority allocation strategies are used to characterize the resource allocation priority ranking rules for different types of containerized tasks, so as to achieve differentiated matching of resource supply and demand. For example, real-time tasks (such as streaming data processing tasks) have higher priority than non-real-time tasks (such as batch processing tasks), and high-priority business tasks have higher priority than low-priority test tasks.

[0039] In practical implementation, the resource requirement assessment module can obtain resource requirement assessment data from the data storage module and real-time monitoring data from the real-time monitoring module, and then merge historical monitoring data with real-time monitoring data. Furthermore, by combining the differences in resource supply and demand, the characteristics of task execution stages (such as startup, computation, I / O, and idle stages), and task attributes (such as real-time / non-real-time, computation-intensive / I / O-intensive), multi-dimensional correlation analysis is performed on the merged data. Finally, resource configuration suggestions and allocation priority strategies for each stage of the task's entire lifecycle are output and transmitted to the dynamic resource allocation module. This achieves dynamic resource adaptation and effectively improves cluster resource utilization and task execution efficiency.

[0040] Step S30: Based on the improved fair scheduling algorithm and logical resource partitioning mechanism, the dynamic resource allocation module formulates the optimal scheduling strategy by combining the resource configuration suggestions, the allocation priority strategy and the real-time load status of each logical resource sub-pool. The module then uses the cluster management interface to achieve precise allocation of resources and tasks and dynamically reclaims and redistributes idle resources throughout the entire lifecycle of the task.

[0041] Among them, the improved fair scheduling algorithm is used to characterize the intelligent scheduling algorithm based on the traditional fair scheduling algorithm, which introduces dynamic factors such as task priority, resource supply and demand differences, task stage characteristics and historical data feedback, and achieves both fairness and efficiency through dynamic weight adjustment and staged adaptation, providing core support for dynamic resource allocation.

[0042] The logical resource partitioning mechanism is used to characterize the overall physical resources of the cluster into multiple isolated logical resource sub-pools based on a two-dimensional matching logic of task attributes (such as compute-intensive, memory-intensive, I / O-intensive, real-time / non-real-time) and resource types (CPU, memory, network bandwidth, storage I / O quota, etc.) without changing the physical resource topology of the cluster. It supports dynamic quota adjustment and enables on-demand supply and isolated scheduling of tasks, including dedicated resource pools for CPU / memory / I / O intensive and real-time tasks.

[0043] Real-time load status is used to characterize the dynamic quantitative status of the logical resource sub-pool at the current moment, including resource utilization intensity (such as CPU utilization, memory utilization, and IO bandwidth utilization), remaining supply capacity (such as remaining CPU cores, available memory capacity, and idle IO quota), and task queue pressure (such as the number of tasks to be scheduled, task priority distribution, and total resource demand to be allocated), providing real-time decision-making basis for improved fair scheduling algorithms.

[0044] The optimal scheduling strategy is used to characterize a dynamic resource allocation and task scheduling scheme based on the real-time load of sub-pools, task requirements and priorities, and with the goal of balancing fairness and efficiency, so as to achieve dual optimization of resource utilization and task execution efficiency.

[0045] Idle resources are used to characterize various reclaimable resources in the cluster that have been allocated to tasks or logical resource sub-pools but have not been effectively utilized, exceed task requirements, or are temporarily idle during the entire lifecycle of a task. These include resources that are not released in a timely manner after a task is completed (such as memory, CPU cores, network connections, storage IO quotas, etc.), resources that are redundantly allocated to tasks during low-load phases (such as IO waiting phases, data transmission gaps) (such as CPU cores, memory capacity, network bandwidth in idle states that exceed actual requirements), resources that are continuously occupied during task pauses, retries, or abnormal suspensions, and idle resources in logical resource sub-pools that have not been allocated to any tasks (such as the remaining CPU quota and unused storage IO bandwidth in long-term low-load sub-pools).

[0046] These idle resources have the common characteristics of being "dynamically recyclable and reallocated". Through the real-time detection and recycling mechanism of the dynamic resource allocation module, they can be reallocated to tasks with urgent resource needs or logical resource sub-pools with tight loads, thereby avoiding resource waste and improving the overall resource utilization of the cluster.

[0047] In its implementation, the dynamic resource allocation module receives resource configuration suggestions and allocation priority strategies for each stage of the task's entire lifecycle from the resource demand assessment module, and obtains the real-time load status of each logical resource sub-pool (including resource utilization, remaining quota, task queue pressure, etc.). Then, based on the improved fair scheduling algorithm and logical resource partitioning mechanism, it dynamically formulates the optimal scheduling strategy by comprehensively considering the resource configuration suggestions, allocation priority strategies, and the real-time load status of each logical resource sub-pool. Subsequently, the cluster management interface enables precise matching of tasks and logical resource sub-pools and dynamic allocation of resource quotas. It also monitors resource usage status in real time throughout the task's entire lifecycle, dynamically reclaims idle resources generated during task execution, and reallocates them to urgently needed tasks or logical resource sub-pools with low load based on the optimal scheduling strategy, forming a dynamic closed loop of "allocation-detection-reclaiming-reallocation".

[0048] Step S40: When a task execution failure is detected, the exception handling module initiates an exception root cause analysis mechanism that links resource usage status with task execution results. Combined with the real-time monitoring data, a linked diagnosis is performed to verify time synchronization and demand adaptability. If it is determined that the task failure is caused by a resource allocation problem, an adaptive response mechanism is triggered. The dynamic resource allocation module adjusts the resource configuration of the task, updates the optimal scheduling strategy, and restarts the task instance to achieve task recovery.

[0049] Among them, the anomaly root cause analysis mechanism is used to characterize the intelligent scheduling system of big data tasks. By linking the entire life cycle data of resource usage and task execution result data, and combining multi-dimensional tracing and correlation analysis, it is an intelligent analysis mechanism that accurately locates the root causes of task anomalies (such as timeouts, failures, and resource exhaustion) and resource scheduling imbalances (such as supply and demand mismatches and overloads), supporting the closed-loop optimization of the system.

[0050] This mechanism enables precise and rapid identification of the root causes of anomalies through the linkage of resources and tasks and cross-validation of historical and real-time data, providing data support for dynamic system optimization and further enhancing the stability, reliability, and adaptability of the big data task scheduling system.

[0051] In its implementation, the exception handling module monitors task execution status in real time. When a task failure is detected, an exception root cause analysis mechanism is automatically initiated. Based on real-time monitoring data, task execution logs, and historical exception cases, a "resource-task" linkage diagnosis is conducted. Specifically, the root cause is accurately determined by verifying two core dimensions: first, data timeline synchronization, ensuring that resource usage data matches the time of the task execution stage to avoid analysis bias caused by data misalignment; second, resource demand adaptability, verifying the matching degree between the allocated resource type / quota and the requirements of the task stage to identify supply and demand mismatch issues. If the task failure is confirmed to be caused by resource allocation issues (such as insufficient resource supply, mismatched resource types, scheduling priority deviation, or resource preemption due to sub-pool overload), an adaptive response mechanism will be triggered immediately: the dynamic resource allocation module will adjust the resource configuration of the task throughout its entire lifecycle (such as supplementing the number of CPU cores, optimizing memory quotas, and switching to a suitable logical resource sub-pool) in a targeted manner based on the results of the anomaly root cause analysis, and update the optimal scheduling strategy simultaneously (such as adjusting the priority weight of the task and optimizing the sub-pool resource quota allocation rules), and then restart the task instance to achieve rapid recovery.

[0052] It is important to note that if a task failure is determined to be due to reasons other than resource allocation issues (such as task code defects, dependent service failures, abnormal data input, etc.), only the exception details of the task (including execution logs, root cause analysis reports, and failure context data) will be recorded in the data storage module. At the same time, a precise exception alarm (including the task's unique identifier, failure timestamp, and preliminary root cause determination results) will be pushed to the operations and maintenance personnel. This will avoid ineffective resource scheduling operations from consuming system computing power, ensure resource scheduling efficiency and targeted fault handling, and further improve system stability and operations and maintenance response efficiency.

[0053] In this embodiment, by constructing a fully automated closed-loop system that integrates real-time monitoring, accurate evaluation, dynamic allocation, and intelligent fault tolerance, the system precisely addresses core issues such as lagging resource monitoring, low allocation accuracy, reliance on manual scheduling, and weak targeted anomaly handling. This achieves the technical effects of improved resource utilization, optimized task execution efficiency, reduced operation and maintenance costs, and enhanced system stability.

[0054] Based on the foregoing embodiments, a second embodiment of the intelligent scheduling method for big data tasks of the present invention is proposed. In this embodiment, step S10 may include the following sub-steps S101~S103:

[0055] Sub-step S101 involves collecting key metrics of containerized task execution in real time through the task status acquisition unit.

[0056] The real-time monitoring module consists of a task status acquisition unit, a resource status acquisition unit, and a data preprocessing unit. Both the task status acquisition unit and the resource status acquisition unit establish a bidirectional real-time data transmission channel with the data preprocessing unit via the Socket communication protocol. The data preprocessing unit synchronizes the preprocessed structured real-time monitoring data to the resource demand assessment module and the data storage module via a high-speed data bus.

[0057] In practical implementation, JProfiler technology, deployed within the execution container of each big data computing task by the task status acquisition unit, can be used to collect key metrics of containerized task execution in real time.

[0058] Sub-step S102 involves collecting the hardware resource status of cluster nodes in real time through the resource status acquisition unit.

[0059] In a practical implementation, the hardware resource status of cluster nodes can be collected in real time through kernel-mode probes deployed on each node of the cluster by the resource status acquisition unit.

[0060] Sub-step S103 involves cleaning, standardizing, and extracting features from the key indicators and hardware resource status using a data preprocessing unit, and outputting structured real-time monitoring data.

[0061] In practical implementation, the data preprocessing unit can use the distributed streaming computing framework Apache Flink (Flink for short, which has high throughput and low latency real-time data processing capabilities) to clean, standardize the format and extract features of the key indicators and hardware resource status collected above, generate and output structured real-time monitoring data, and then synchronize it to the resource demand assessment module and data storage module through a high-speed data bus to provide standardized data input for the downstream resource demand assessment module.

[0062] In this embodiment, JProfiler technology is used to collect key indicators of the task at high frequency, kernel-mode probes are used to perceive cluster hardware resources in real time, and Flink streaming preprocessing is used to optimize data quality. This solves the problems of insufficient real-time monitoring and low data quality, and provides standardized data support with high real-time performance and high quality for subsequent accurate resource assessment and dynamic scheduling.

[0063] Based on the foregoing embodiments, a third embodiment of the intelligent scheduling method for big data tasks of the present invention is proposed. In this embodiment, step S20 may include the following sub-steps S201~S202:

[0064] Sub-step S201: The historical monitoring data and the real-time monitoring data are obtained through the data integration unit, and then classified, summarized and integrated according to data dimensions to generate a unified integrated monitoring dataset.

[0065] The resource demand assessment module consists of a data integration unit and a resource demand suggestion unit. Its core is to integrate historical monitoring data from the data storage module with real-time monitoring data from the real-time monitoring module, and combine the differences in resource supply and demand, the characteristics of the task execution stage (such as the startup stage, computation stage, IO stage, and idle stage), the task attributes (such as computation-intensive / IO-intensive, real-time / non-real-time), and the supply and demand matching of the resources requested by the task and the resources actually used. After multi-dimensional correlation analysis, it outputs resource configuration suggestions and allocation priority strategies for each stage of the task's entire life cycle.

[0066] The integrated monitoring dataset is used to represent the unified format fusion dataset formed after the integration unit performs fusion processing (including data alignment, redundancy removal, and time axis synchronization) on the historical monitoring data of the data storage module and the real-time monitoring data of the real-time monitoring module. This dataset covers the cluster resource status, task execution status, and resource allocation and usage, providing a full-dimensional and high-quality data foundation for the multi-dimensional correlation analysis of the resource demand analysis and suggestion unit.

[0067] In practical implementation, the data integration unit can obtain historical monitoring data (including resource application records, actual usage records, and execution effect data for similar tasks) from the data storage module and real-time monitoring data (including the current cluster resource pool supply status, resource application information for the current task, and real-time usage data of allocated resources) from the real-time monitoring data module. Then, the data integration unit can integrate the two types of data into a unified format integrated monitoring dataset through data alignment, time axis synchronization, redundancy removal, and dimension classification.

[0068] Sub-step S202 involves using the resource demand suggestion unit to perform multi-dimensional correlation analysis on the integrated monitoring dataset, combining the characteristics of the task execution stage and the differences in resource supply and demand, and outputting resource configuration suggestions and allocation priority strategies for each stage of the task's entire lifecycle.

[0069] In practical implementation, the resource requirement suggestion unit can combine the characteristics of the task execution stage, task attributes, and differences in resource supply and demand. Through analysis links such as the correlation between historical and real-time data and the mapping between task characteristics and resource requirements, multi-dimensional correlation analysis can be carried out on the integrated monitoring dataset. Finally, resource configuration suggestions and allocation priority strategies for each stage of the task's entire life cycle can be output.

[0070] In this embodiment, by integrating historical data on similar tasks with real-time status data of current tasks, and comprehensively combining the dynamic characteristics, attributes, and actual supply and demand of resources of tasks, a precise quantitative analysis is conducted. This effectively solves the core shortcomings of cluster resource allocation, which relies on static configuration and does not fully utilize historical experience and real-time dynamics. It avoids pain points such as low allocation accuracy, resource supply and demand mismatch (such as memory redundancy and insufficient CPU for computationally intensive tasks), and low resource utilization. This provides a precise decision-making basis for the subsequent dynamic resource allocation module that combines historical reference value and real-time adaptability, ultimately achieving dynamic adaptation and efficient utilization of cluster resources.

[0071] Based on the foregoing embodiments, a fourth embodiment of the intelligent scheduling method for big data tasks of the present invention is proposed. In this embodiment, step S30 may include the following sub-steps S301~S303:

[0072] In sub-step S301, the resource pool management unit collects the original load data of each logical resource sub-pool in real time according to the logical resource partitioning mechanism. After preprocessing by filtering instantaneous peak values ​​and removing invalid data, the real-time load status of each logical resource sub-pool is generated.

[0073] The dynamic resource allocation module consists of a resource scheduling engine and a resource pool management unit. The resource scheduling engine implements the core scheduling logic based on an improved fair scheduling algorithm, breaking through the static limitations of traditional fair scheduling and possessing dynamic weight adjustment and phased adaptation capabilities. The resource pool management unit, based on a logical partitioning mechanism, divides cluster resources into multiple functionally dedicated logical resource sub-pools (such as CPU-intensive task resource pools, memory-intensive task resource pools, and real-time task-specific resource pools) through virtualization partitioning technology, supporting dynamic scaling and isolated scheduling of sub-pool resources.

[0074] The resource scheduling engine establishes a two-way data interaction link with the resource demand assessment module and the resource pool management unit: On the one hand, the resource scheduling engine receives resource configuration suggestions and allocation priority strategies for each stage of the task's entire lifecycle from the resource demand assessment module, serving as the core basis for scheduling decisions. On the other hand, the resource scheduling engine obtains real-time load status (including core dimensions such as resource utilization, remaining quota, and task queue pressure) of each logical resource sub-pool from the resource pool management unit, thus grasping the dynamics of cluster resource supply.

[0075] The resource scheduling engine synchronizes the optimal scheduling strategy to the resource pool management unit, supporting the dynamic adjustment of sub-pool resource quotas (such as cross-sub-pool resource sharing, sub-pool expansion / contraction), forming a closed-loop interaction of "data collection → decision making → execution feedback".

[0076] In practical implementation, the resource pool management unit can collect the raw load data of each logical resource sub-pool in real time through the cluster resource monitoring interface according to the logical resource partitioning mechanism. After filtering out instantaneous peak interference, removing abnormal and invalid data, and standardizing the data format, a standardized real-time load status of each logical resource sub-pool is generated. The real-time load status includes three dimensions: the remaining resource amount of each logical resource sub-pool, the utilization rate of allocated resources, and the characteristics of the task queue (queue length, total resource requirements of tasks to be scheduled, and task priority distribution). Subsequently, the real-time load status of each logical resource sub-pool is synchronously fed back to the resource scheduling engine, providing accurate and reliable dynamic load data support for the resource scheduling engine to formulate the optimal scheduling strategy based on resource configuration suggestions.

[0077] Sub-step S302 involves using the resource scheduling engine to perform standardized processing on the resource configuration suggestions, allocation priority strategies, and real-time load status of each logical resource sub-pool, including unifying resource measurement units, filling in missing fields, and removing outliers, to generate a standardized dataset.

[0078] In practical implementation, the resource scheduling engine can receive the real-time load status of each logical resource sub-pool from the resource management pool unit, and synchronously obtain the resource configuration suggestions and allocation priority strategies for the entire lifecycle of the task output by the resource demand assessment module. Then, for the above three types of core data (resource configuration suggestions, allocation priority strategies, and real-time load status of each logical resource sub-pool), a standardized processing flow is executed to unify resource measurement units, fill in missing fields, and remove outliers. Finally, a standardized dataset that meets the requirements of multi-dimensional correlation analysis is generated, providing data support with consistency, completeness, and accuracy for the dynamic formulation of subsequent optimal scheduling strategies.

[0079] Sub-step S303 involves the resource scheduling engine using the improved fair scheduling algorithm and the standardized dataset to sequentially complete task and logical resource sub-pool adaptation, priority sorting and fairness calibration, resource supply and demand balance calculation, and finally dynamically formulate the optimal scheduling strategy.

[0080] In the specific implementation, the resource scheduling engine can use the improved fair scheduling algorithm, with the aforementioned standardized dataset as the core input, and execute the process step by step according to the progressive logic of "task-sub-pool adaptation and matching → priority sorting and fairness calibration → resource supply and demand balance calculation," ultimately dynamically determining the optimal scheduling strategy. The specific process is as follows: Task-Subpool Adaptation Matching: Combining task attributes with the functional positioning and real-time load status of each logical resource subpool, the system selects target logical resource subpools that are highly compatible with the current task's resource requirements through precise matching of "task requirement characteristics - subpool resource supply," thereby avoiding resource waste caused by mismatch between cross-type tasks and subpools from the source.

[0081] Priority ranking and fairness calibration: First, based on the allocation priority strategy output by the resource demand assessment module, the tasks to be scheduled in the target sub-pool are initially prioritized (high-priority tasks are scheduled first). Then, a dynamic fairness weight is introduced through an improved fair scheduling algorithm (combining task waiting time, historical scheduling fairness, and sub-pool load balancing objectives) to calibrate the initial ranking result. For example, while ensuring the core scheduling rights of high-priority tasks, the scheduling weight of ordinary tasks that have been waiting for a long time is appropriately increased, effectively avoiding the problem of "ordinary task starvation caused by priority preemption," and achieving the ranking goal of "efficiency first, while taking fairness into account."

[0082] Resource supply and demand balancing calculation: Based on resource supply and demand difference data in a standardized dataset (such as remaining resource quotas in sub-pools and peak resource demand at each stage of the task's lifecycle), combined with task priority weights and sub-pool load saturation, a quantitative matching calculation of supply and demand gaps (such as a supply and demand gap matching algorithm) is used to accurately determine the optimal resource allocation quotas (such as the number of CPU cores, memory capacity, IO bandwidth, and network connections) for each task. This calculation process ensures that the allocation results meet the core needs of each stage of the task, strictly controls the allocation within the sub-pool's resource capacity limit, and reserves 5%-10% of the total sub-pool resources as elastic resources to cope with sudden loads (such as temporary surges in task demand or fluctuations in node resources).

[0083] The entire process achieves an upgrade of the scheduling strategy from "static rules" to "dynamic adaptation" through the deep integration of the improved fair scheduling algorithm and the standardized dataset. This ensures that the final optimal scheduling strategy is targeted (precise matching of tasks and sub-pools, adapting to task types and resource supply characteristics), fair (balancing the efficiency of priority tasks and the rights of ordinary tasks, avoiding scheduling imbalance), and feasible (quantitative balance of supply and demand, reserving elastic resources to ensure system stability). This provides core algorithm execution support for the accurate allocation, dynamic recycling, and efficient utilization of cluster resources.

[0084] Furthermore, in one embodiment, step S30 may further include the following sub-steps S304~S307: In sub-step S304, the resource scheduling engine performs container creation, resource binding, and task scheduling operations through the cluster management interface according to the optimal scheduling strategy, accurately matching the target resources and tasks in the logical resource sub-pool and completing resource allocation.

[0085] In its implementation, after generating the optimal scheduling strategy, the resource scheduling engine executes container creation, resource binding, and task scheduling operations sequentially through the cluster management interface based on this strategy. Subsequently, the resource quotas adapted within the target logical resource sub-pool are precisely and dynamically bound to the tasks, completing on-demand resource allocation. This process strictly adheres to the "task-sub-pool matching relationship, resource allocation quota, and scheduling execution order" explicitly defined in the optimal scheduling strategy, ensuring that tasks receive resource supplies precisely adapted to the needs of each stage of their lifecycle from the startup phase. Simultaneously, leveraging the atomic execution characteristics of container creation, resource binding, and task scheduling, the consistency and reliability of resource allocation are enhanced, effectively avoiding typical intermediate anomalies such as "resource binding failure but container already created" and "task scheduling mismatch with resource quota." Through this design, a solid resource guarantee is laid for the efficient and stable execution of tasks throughout their entire lifecycle.

[0086] In sub-step S305, the resource pool management unit updates the load status of each logical resource sub-pool in real time based on the resource allocation execution result of the dynamic resource allocation module, and synchronizes it to the resource scheduling engine.

[0087] In the specific implementation, the resource pool management unit updates the load status (including resource utilization, remaining quota, task queue information, etc.) of each logical resource sub-pool in real time based on the resource allocation execution results of the dynamic resource allocation module (such as successful resource quota binding and task scheduling completion). The updated real-time load status is then synchronously fed back to the resource scheduling engine in real time, providing the latest load data support for the resource scheduling engine to adjust the optimal scheduling strategy and achieve dynamic balance between resource supply and demand.

[0088] Sub-step S306 involves using the resource scheduling engine to monitor the usage status of allocated resources in real time throughout the entire lifecycle of the task, and triggering a recycling mechanism for resources that have reached the idle threshold, so that the recycled idle resources are returned to the corresponding logical resource sub-pool.

[0089] In its implementation, the resource scheduling engine continuously monitors the usage status of allocated resources throughout the entire task lifecycle. Based on preset idle resource thresholds (e.g., CPU utilization below 10% for 5 consecutive minutes, memory usage below 20% for 5 consecutive minutes), an elastic reclamation mechanism is automatically initiated for resources reaching these thresholds. After reclamation, idle resources are released back to the corresponding logical resource sub-pool and marked as "allocatable," ready for rescheduling by subsequent new tasks or tasks with urgent resource needs. This mechanism precisely matches tasks with sub-pools and coordinates with the cluster resource management interface's scheduling execution function, enabling real-time reclamation and reallocation of idle resources (such as redundant CPUs during low-load task phases or memory not released after execution). This ensures dynamic on-demand supply of cluster resources, eliminating idle waste and significantly improving overall resource utilization.

[0090] In sub-step S307, the resource scheduling engine dynamically adjusts the subsequent scheduling strategy based on the updated logical resource sub-pool load status synchronized by the resource management pool unit, combined with the improved fair scheduling algorithm. The reclaimed idle resources are then combined with resource supply and demand difference data and allocation priority strategy to prioritize the reallocation to tasks with urgent resource needs, thereby realizing dynamic recycling of resources.

[0091] In practical implementation, the resource scheduling engine can dynamically adjust the subsequent optimal scheduling strategy based on the updated load status of the logical resource sub-pool synchronized by the resource pool management unit, combined with the improved fair scheduling algorithm. Subsequently, the reclaimed idle resources can be combined with resource supply and demand difference data and allocation priority strategy to be preferentially reallocated to high-priority tasks with resource demand or scheduled tasks in the logical resource sub-pool with tight load, realizing the dynamic recycling of cluster resources through "allocation-reclaiming-reallocation", further improving the accuracy of resource configuration and overall utilization.

[0092] In this embodiment, the dynamic resource allocation module employs a collaborative architecture of "improved fair scheduling algorithm + logical resource pooling management" to specifically address the core pain points of cluster resource allocation, such as reliance on static manual configuration, lack of automatic adaptation mechanisms, and intense competition for resources across different task types. Through the deep coupling of logical resource sub-pool isolation and the improved fair algorithm, a closed-loop management system is achieved for the entire lifecycle of resources: "precise allocation on demand - intelligent recycling of idle resources - dynamic replenishment of gaps." This ultimately results in overall cluster load balancing, maximized resource utilization, fair scheduling, and stable task execution, providing core resource scheduling support for the efficient and stable operation of big data tasks.

[0093] Based on the foregoing embodiments, a fifth embodiment of the intelligent scheduling method for big data tasks of the present invention is proposed. In this embodiment, step S40 may include the following sub-steps S401~S403:

[0094] In sub-step S401, the anomaly detection unit responds to the task execution failure event, receives the real-time monitoring data through the high-speed data bus, and extracts resource usage status data and task execution status data after data parsing; wherein, the resource usage status data includes the task's CPU utilization rate, memory utilization rate, resource allocation quota, and resource load fluctuation; the task execution status data includes the task failure flag, failure timestamp, and failure log details.

[0095] The anomaly detection module consists of an anomaly detection unit and an optimization and restart unit. Based on a closed-loop architecture of "detection-analysis-control," it achieves accurate identification of task execution anomalies, preliminary root cause determination, and adaptive recovery processing.

[0096] The anomaly detection unit employs a "resource usage status - task execution result" linkage detection mechanism. It receives resource usage data and task execution status data generated from real-time monitoring data parsing via a high-speed data bus (both types of data are continuously collected and standardized by the real-time monitoring module throughout its lifecycle). This unit aligns and cross-references the execution timelines of these two types of data to quickly identify "task execution anomalies triggered by abnormal resource usage" (such as task timeouts due to continuous CPU overload, task crashes due to memory leaks, and task data read / write failures due to IO bandwidth exhaustion). This provides complete raw data support for accurate determination by the subsequent anomaly root cause analysis mechanism, and the detection results are directly output to the optimization and restart unit.

[0097] The optimization and restart unit establishes a two-way collaborative control link with the dynamic resource allocation module, and simultaneously establishes a real-time control connection with the task execution container through the cluster resource management interface. Specific interaction and control logic: ① Data input level: Obtain the resource allocation record of the target task (including resource allocation quota, optimal scheduling strategy execution log, and real-time load data of the logical resource sub-pool) from the dynamic resource allocation module. Combined with the detection results of the anomaly detection unit, preliminarily determine whether the task anomaly is caused by resource-related issues such as resource allocation imbalance, insufficient resource supply, or sub-pool overload.

[0098] ② Control output level: If the anomaly is determined to be resource-related, a resource adjustment instruction (such as expanding CPU / memory quota, switching to an appropriate logical resource sub-pool, or increasing task scheduling priority weight) is issued to the dynamic resource allocation module, and the safe restart operation of the task instance is executed through the container control connection; if the anomaly is determined to be non-resource-related, only the anomaly details (including detection results, preliminary root cause determination, task context data, and failure timestamp) are synchronized to the data storage module for persistent storage, and the operation and maintenance personnel are triggered to push accurate anomaly alarms.

[0099] Resource usage status data is real-time, continuous, and dynamically quantified monitoring data collected by the real-time monitoring module throughout the entire task lifecycle to characterize the usage of allocated resources. It reflects the resource consumption characteristics of the task and the load change trends of the logical resource sub-pool. Specifically, it includes: resource utilization indicators (task CPU utilization, memory utilization, IO bandwidth utilization), resource allocation benchmarks (task resource allocation quota), and load fluctuation characteristics (resource load fluctuation curves, fluctuation frequency, and peak / valley data). This data provides core quantitative basis for the phased resource demand calculation of the resource assessment module, the formulation of optimal scheduling strategies by the dynamic resource allocation module, and the identification of resource-related anomalies by the anomaly detection unit.

[0100] Task execution status data is used to characterize the status indicators and anomaly details of the execution process continuously monitored and measured by the real-time monitoring module throughout the entire lifecycle of the task, from submission to completion. This data reflects the task execution progress and fault occurrence characteristics. Specifically, it includes: execution result identifiers (task execution failure identifier, execution success identifier, execution in progress identifier), fault time characteristics (task failure timestamp), and anomaly details (failure log details, anomaly stack log), as well as execution progress quantification (task progress percentage). This data provides crucial status support for the anomaly detection unit's task execution anomaly triggering, the anomaly root cause analysis mechanism's fault type determination, and the scheduling restart unit's adaptive recovery operations.

[0101] In practical implementation, the anomaly detection unit can respond to task execution failure events in real time. It adopts a "resource usage status - task execution result" linkage detection mechanism and receives resource usage data and task execution status data continuously collected by the real-time monitoring module throughout the entire life cycle and generated through standardized parsing via a high-speed data bus.

[0102] Sub-step S402 involves using the anomaly detection unit to perform a linkage analysis on the resource usage status data and the task execution status data, and to complete the time synchronization and demand adaptability verification in multiple dimensions. Specifically, it involves determining the time overlap between the task failure time and the resource anomaly state, verifying whether the resource anomaly occurred before or simultaneously with the task failure event, and clarifying the temporal correlation between the two. Furthermore, it involves comparing the current resource allocation amount with the resource configuration suggestions output by the resource demand assessment module to verify whether the resource allocation is below the task demand lower limit, whether there is a resource allocation imbalance, or whether there is insufficient available resources due to resource competition.

[0103] In the specific implementation, the anomaly detection unit can perform timeline alignment and cross-validation on the two types of data mentioned above to quickly identify "task execution anomalies triggered by abnormal resource usage". The specific verification logic is divided into two layers: ① Timing correlation judgment: Quantitatively calculate the time overlap between the task failure time and the abnormal resource state to verify whether the resource anomaly occurred before or simultaneously with the task failure event, and clarify the temporal causal relationship between the resource anomaly and the task failure; ② Resource allocation rationality verification: Compare the current task's resource allocation quota with the task's full lifecycle resource configuration suggestions output by the resource demand assessment module in multiple dimensions, focusing on verifying whether the resource allocation value is lower than the task's minimum requirement, whether there is an imbalance in the resource allocation structure (such as sufficient CPU quota but insufficient memory quota), and whether the actual available resources of the task are lower than the allocated quota due to overload of the logical resource sub-pool or cross-task resource competition. The anomaly detection unit forms a complete detection result through the above two-layer verification, providing raw data support including timing correlation data, resource allocation verification data, and abnormal scenario characteristics for the accurate determination of the subsequent anomaly root cause analysis mechanism. The detection result is directly output to the optimization and restart unit.

[0104] Furthermore, in one embodiment, step S40 may further include the following sub-steps S403-S406: In sub-step S403, after the anomaly detection unit determines that the task failure was caused by a resource allocation problem, the determination result of the failure caused by the resource allocation problem and the anomaly details are output to the optimization and restart unit.

[0105] In the specific implementation, after the anomaly detection unit determines that the task failure is caused by a resource allocation problem based on the detection results of the aforementioned two-layer verification, it outputs the determination result and complete anomaly details (including time-series correlation data, resource allocation verification data, and anomaly scene characteristics) to the optimization and restart unit.

[0106] In sub-step S404, the optimization and restart unit sends a bidirectional request to the dynamic resource allocation module to obtain the historical resource allocation record of the task and submit targeted resource configuration adjustment requirements.

[0107] In practical implementation, the optimization and restart unit can send two types of targeted requests to the dynamic resource allocation module through the bidirectional collaborative control link established with it: ① Initiating a data query request to obtain the historical resource allocation record of the target task (including historical resource allocation quota, optimal scheduling strategy execution record, and peak / valley data of resource usage); ② Sending a configuration scheduling request to submit resource configuration adjustment requirements for the task (including quota expansion, allocation structure optimization, and sub-pool switching suggestions).

[0108] In sub-step S405, the dynamic resource allocation module combines the historical resource allocation records of the task, the real-time load status of the current logical resource sub-pool, and the resource configuration suggestions to complete the precise adjustment of the resource configuration of the task, and simultaneously update the optimal scheduling strategy. Then, the resource adjustment results are fed back to the optimization and restart unit.

[0109] In the specific implementation, the dynamic resource allocation module can combine the historical resource allocation records of the target task, the real-time load status of the current logical resource sub-pool, and the task's full lifecycle resource configuration suggestions output by the resource demand assessment module to accurately adjust the resource configuration of the task through an improved fair scheduling algorithm (such as expanding CPU / memory quotas, optimizing resource allocation structure, and switching to a lower-load adaptive logical resource sub-pool), and simultaneously iterate and update the optimal scheduling strategy. Subsequently, the resource adjustment results, including the adjusted resource quotas, sub-pool allocation results, and strategy update identifiers, can be fed back to the tuning and restart unit.

[0110] In sub-step S406, the resource adjustment result is received through the optimization and restart unit. After confirming that the adjusted resource configuration meets the task requirements and does not exceed the resource limit of the current logical resource sub-pool, a restart command is sent to the task execution container through the cluster management interface so that the task can be re-executed in a sufficient and suitable resource environment.

[0111] In its implementation, after receiving the resource adjustment results, the optimization and restart unit performs dual compliance checks on the adjusted resource configuration: ① checking whether it meets the lower limit of resource requirements at each stage of the task's entire lifecycle; ② checking whether it does not exceed the resource capacity limit of the current logical resource sub-pool. After passing the checks, the optimization and restart unit sends a task restart command to the task execution container through the cluster resource management interface, enabling the task to be re-executed in a resource-sufficient and precisely adapted-to-requirements operating environment.

[0112] In this embodiment, the anomaly handling module, relying on the collaborative linkage of the anomaly detection unit and the optimization and restart unit, realizes a processing mechanism for anomaly data collection, correlation detection, preliminary root cause determination, resource optimization, and task restart. This effectively improves the efficiency of task execution anomaly identification and adaptive recovery capability, reduces the interference of resource-related anomalies on task execution from the source, and effectively reduces the task failure rate caused by resource imbalance and insufficient resource supply. This provides a core guarantee for the overall stability and reliable operation of the intelligent scheduling system for big data tasks.

[0113] Based on the same inventive concept, the sixth embodiment of this invention also provides an intelligent scheduling system for big data tasks, corresponding to the intelligent scheduling method for big data tasks in the foregoing embodiments. Since the principle by which the system in the sixth embodiment solves the problem is similar to the intelligent scheduling method for big data tasks in the foregoing embodiments, the implementation of the system can refer to the implementation of the method; repeated details will not be elaborated further. Please refer to... Figure 2 The present invention provides an intelligent scheduling system for big data tasks, the system comprising:

[0114] The real-time monitoring module 10 is used to collect key indicators of containerized task execution and the hardware resource status of cluster nodes in real time, and generate structured real-time monitoring data after preprocessing. The resource demand assessment module 20 is used to integrate historical monitoring data and real-time monitoring data, and to perform multi-dimensional correlation analysis by combining the differences in resource supply and demand and the characteristics of the task execution stage, and output resource configuration suggestions and allocation priority strategies for the entire life cycle of the task. The dynamic resource allocation module 30 is used to formulate the optimal scheduling strategy based on the improved fair scheduling algorithm and logical resource partitioning mechanism, combined with the resource configuration suggestions, the allocation priority strategy and the real-time load status of each logical resource sub-pool, and to realize the precise allocation of resources and tasks through the cluster management interface, and to dynamically reclaim and redistribute idle resources throughout the entire life cycle of the task. The exception handling module 40 is used to initiate an exception root cause analysis mechanism that links resource usage status with task execution results when a task execution failure is detected. It combines the real-time monitoring data to carry out linked diagnosis to verify time synchronization and demand adaptability. If it is determined that the task failure is caused by a resource allocation problem, an adaptive response mechanism is triggered. The dynamic resource allocation module adjusts the resource configuration of the task, updates the optimal scheduling strategy, and restarts the task instance to achieve task recovery.

[0115] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned intelligent scheduling method for big data tasks.

[0116] Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 3 The general will label all buses as bus systems.

[0117] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0118] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0119] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The intelligent scheduling method for big data tasks provided in this embodiment of the invention can be included in the application program 4022.

[0120] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the intelligent scheduling method for big data tasks provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0121] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0122] In summary, this invention, by constructing a fully automated closed-loop system that integrates real-time monitoring, precise evaluation, dynamic allocation, and intelligent fault tolerance, precisely addresses core issues such as lagging resource monitoring, low allocation accuracy, reliance on manual scheduling, and weak targeted anomaly handling. This results in improved resource utilization, optimized task execution efficiency, reduced maintenance costs, and enhanced system stability.

[0123] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An intelligent scheduling method for big data tasks, characterized in that, An intelligent scheduling system for big data tasks, comprising a real-time monitoring module, a resource demand assessment module, a dynamic resource allocation module, an exception handling module, and a data storage module, wherein the method includes: The real-time monitoring module collects key indicators of containerized task execution and the hardware resource status of cluster nodes in real time, and generates structured real-time monitoring data after preprocessing. By integrating historical and real-time monitoring data through the resource demand assessment module, and combining the differences in resource supply and demand with the characteristics of the task execution stage, multi-dimensional correlation analysis is performed to output resource configuration suggestions and allocation priority strategies for the entire task lifecycle. The dynamic resource allocation module, based on the improved fair scheduling algorithm and logical resource partitioning mechanism, combined with the resource configuration suggestions, the allocation priority strategy and the real-time load status of each logical resource sub-pool, formulates the optimal scheduling strategy, and realizes accurate allocation of resources and tasks through the cluster management interface, and dynamically reclaims and redistributes idle resources throughout the entire life cycle of the task. When a task execution failure is detected, an anomaly root cause analysis mechanism that links resource usage status with task execution results is initiated through the anomaly handling module. Combined with the real-time monitoring data, a linked diagnosis is carried out to verify time synchronization and demand adaptability. If it is determined that the task failure is caused by a resource allocation problem, an adaptive response mechanism is triggered. The resource configuration of the task is adjusted through the dynamic resource allocation module, the optimal scheduling strategy is updated, and the task instance is restarted to achieve task recovery.

2. The method according to claim 1, characterized in that, The real-time monitoring module includes a task status acquisition unit, a resource status acquisition unit, and a data preprocessing unit. The real-time monitoring module acquires key indicators of containerized task execution and the hardware resource status of cluster nodes in real time, and generates structured real-time monitoring data through preprocessing, including: Key metrics for containerized task execution are collected in real time through the task status acquisition unit. The hardware resource status of cluster nodes is collected in real time through the resource status acquisition unit; The data preprocessing unit cleans, standardizes, and extracts features from the key indicators and hardware resource status to output structured real-time monitoring data.

3. The method according to claim 1, characterized in that, The resource demand assessment module includes a data integration unit and a resource demand suggestion unit. The module integrates historical and real-time monitoring data, and performs multi-dimensional correlation analysis based on resource supply and demand differences and task execution stage characteristics. It outputs resource allocation suggestions and priority strategies for the entire task lifecycle, including: The data integration unit acquires historical monitoring data and real-time monitoring data, and categorizes and integrates them according to data dimensions to generate a unified integrated monitoring dataset. By combining the resource demand suggestion unit with the characteristics of the task execution stage and the differences between resource supply and demand, a multi-dimensional correlation analysis is performed on the integrated monitoring dataset to output resource configuration suggestions and allocation priority strategies for each stage of the task's entire life cycle.

4. The method according to claim 1, characterized in that, The dynamic resource allocation module includes a resource scheduling engine and a resource pool management unit. The dynamic resource allocation module, based on an improved fair scheduling algorithm and logical resource partitioning mechanism, and combining the resource configuration suggestions, the allocation priority strategy, and the real-time load status of each logical resource sub-pool, formulates an optimal scheduling strategy, including: The resource pool management unit collects the raw load data of each logical resource sub-pool in real time according to the logical resource partitioning mechanism. After preprocessing by filtering instantaneous peak values ​​and removing invalid data, the real-time load status of each logical resource sub-pool is generated. The resource scheduling engine performs standardized processing on the resource configuration suggestions, allocation priority strategies, and real-time load status of each logical resource sub-pool, including unified resource measurement units, completion of missing fields, and removal of outliers, to generate a standardized dataset. The resource scheduling engine, based on the improved fair scheduling algorithm and the standardized dataset, sequentially completes task and logical resource sub-pool adaptation, priority sorting and fairness calibration, resource supply and demand balance calculation, and finally dynamically formulates the optimal scheduling strategy.

5. The method according to claim 4, characterized in that, The method of achieving precise allocation of resources and tasks through a cluster management interface, and dynamically reclaiming and reallocating idle resources throughout the entire task lifecycle, includes: Based on the optimal scheduling strategy, the resource scheduling engine performs container creation, resource binding, and task scheduling operations through the cluster management interface, accurately matching target resources and tasks in the logical resource sub-pool and completing on-demand resource allocation. Based on the resource allocation execution results of the dynamic resource allocation module, the resource pool management unit updates the load status of each logical resource sub-pool in real time and synchronizes it to the resource scheduling engine. The resource scheduling engine monitors the usage status of allocated resources in real time throughout the entire task lifecycle and triggers a recycling mechanism for resources that have reached the idle threshold, returning the recycled idle resources to the corresponding logical resource sub-pool. The resource scheduling engine dynamically adjusts the optimal scheduling strategy based on the updated logical resource sub-pool load status synchronized by the resource pool management unit and the improved fair scheduling algorithm. It also prioritizes the reallocation of reclaimed idle resources to tasks with urgent resource needs, based on resource supply and demand difference data and allocation priority strategy, thereby achieving dynamic recycling of resources.

6. The method according to claim 1, characterized in that, The anomaly handling module includes an anomaly detection unit and an optimization and restart unit. When a task execution failure is detected, the anomaly handling module initiates an anomaly root cause analysis mechanism that links resource usage status with task execution results. This mechanism, combined with real-time monitoring data, performs linked diagnostics to verify time synchronization and requirement adaptability, including: The system responds to task execution failure events through an anomaly detection unit, receives real-time monitoring data through a high-speed data bus, and extracts resource usage status data and task execution status data after data parsing. The anomaly detection unit performs a linkage analysis on the resource usage status data and the task execution status data, and performs time synchronization and demand adaptability verification in multiple dimensions. Specifically, it determines the time overlap between the task failure time and the resource anomaly status to verify whether the resource anomaly occurred before or simultaneously with the task failure event, and clarifies the temporal correlation between the two. In addition, it compares the current resource allocation amount with the resource configuration suggestions output by the resource demand assessment module to verify whether the resource allocation is lower than the task demand lower limit, whether there is a resource allocation imbalance, or whether there is insufficient available resources due to resource competition.

7. The method according to claim 6, characterized in that, If the task failure is determined to be caused by a resource allocation problem, an adaptive response mechanism is triggered. This mechanism adjusts the task's resource configuration, updates the optimal scheduling strategy, and restarts the task instance to restore the task, including: After the anomaly detection unit determines that the task failure was caused by a resource allocation problem, it outputs the determination result of the failure caused by the resource allocation problem and the anomaly details to the optimization and restart unit. The optimization and restart unit sends a two-way request to the dynamic resource allocation module to obtain the historical resource allocation record of the task and submit targeted resource configuration adjustment requirements. By combining the historical resource allocation records of the task, the real-time load status of the current logical resource sub-pool, and the resource configuration suggestions, the dynamic resource allocation module completes the precise adjustment of the resource configuration of the task and updates the optimal scheduling strategy simultaneously. Then, the resource adjustment results are fed back to the optimization and restart unit. After receiving the resource adjustment result through the optimization and restart unit and confirming that the adjusted resource configuration meets the task requirements and does not exceed the resource limit of the current logical resource sub-pool, the unit sends a restart command to the task execution container through the cluster management interface, so that the task can be re-executed in a sufficient and suitable resource environment.

8. An intelligent scheduling system for big data tasks, characterized in that, The system includes: The real-time monitoring module is used to collect key indicators of containerized task execution and the hardware resource status of cluster nodes in real time, and generate structured real-time monitoring data after preprocessing. The resource demand assessment module is used to integrate historical and real-time monitoring data, and combine the differences in resource supply and demand with the characteristics of the task execution stage to conduct multi-dimensional correlation analysis, and output resource configuration suggestions and allocation priority strategies for the entire life cycle of the task. The dynamic resource allocation module is used to formulate the optimal scheduling strategy based on the improved fair scheduling algorithm and logical resource partitioning mechanism, combined with the resource configuration suggestions, the allocation priority strategy and the real-time load status of each logical resource sub-pool, and to realize the precise allocation of resources and tasks through the cluster management interface, and to dynamically reclaim and redistribute idle resources throughout the entire life cycle of the task. The exception handling module is used to initiate an exception root cause analysis mechanism that links resource usage status with task execution results when a task execution failure is detected. It combines the real-time monitoring data to carry out linked diagnosis to verify time synchronization and demand adaptability. If it is determined that the task failure is caused by a resource allocation problem, an adaptive response mechanism is triggered. The dynamic resource allocation module adjusts the resource configuration of the task, updates the optimal scheduling strategy, and restarts the task instance to achieve task recovery.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed, performs the steps of the method according to any one of claims 1 to 7.

Images