Task scheduling optimization analysis method and system based on big data

Abstract

The invention relates to the technical field of computers, in particular to a task scheduling optimization analysis method and system based on big data. The method comprises the following steps: acquiring task related data, task system resource use data and task execution state data in real time in a corresponding distributed big data processing scene among a server, a storage device and a client; performing big data feature mining analysis and task scheduling clustering analysis by using a Hadoop distributed framework to generate task scheduling cluster groups corresponding to similar resource requirements; performing residual execution duration calculation and task emergency accounting on the task calculation nodes corresponding to the distributed big data processing scene to obtain task node scheduling emergency degrees; and performing scheduling priority determination and task scheduling optimization analysis on the corresponding task calculation nodes to generate a task dynamic optimization scheduling scheme. According to the method, the rationality of distributed task resource scheduling can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task scheduling optimization analysis method and system based on big data. Background Art

[0002] In computer systems, task scheduling is a key link in allocating system resources reasonably and ensuring efficient task execution. Traditional task scheduling algorithms mostly make scheduling decisions based on simple factors such as task priority and resource requirements, such as First Come First Served (FCFS) and Shortest Job First (SJF). At the same time, with the advent of the big data era, the scale and complexity of tasks faced by computer systems are growing exponentially, and these traditional algorithms have gradually exposed many shortcomings.

[0003] In cloud computing, data centers and other scenarios, a large number of tasks are submitted at the same time, and the task types are rich and varied, including data processing, machine learning model training, real-time communication, etc. However, traditional scheduling algorithms find it difficult to fully consider the complex characteristics and dynamic changes of distributed tasks between computer devices, such as the real-time requirements of tasks, the volatility of resource usage, and the dependencies between different tasks. There is a lack of in-depth analysis of task computing nodes and system resource usage, and it is impossible to fully tap the value of data, resulting in unreasonable resource allocation and inefficient task execution. Summary of the invention

[0004] Based on this, it is necessary for the present invention to provide a task scheduling optimization analysis method and system based on big data to solve at least one of the above technical problems.

[0005] To achieve the above purpose, a task scheduling optimization analysis method based on big data includes the following steps: Step S1: by collecting task-related data in real time in the corresponding distributed big data processing scenarios between the server, the storage device and the client, including the task submission time, the task deadline, the task type and the task resource requirements; based on the task resource requirements, the system resource usage of the task computing nodes corresponding to the distributed big data processing scenarios is calculated to obtain the task system resource usage data; the task status of the task computing nodes corresponding to the distributed big data processing scenarios is measured to obtain the task execution status data, including the task execution time, the actual usage of the task resources and the task completion status; Step S2: Using the Hadoop distributed framework, perform big data feature mining and analysis on task-related data, task system resource usage data, and task execution status data to obtain a distributed task big data feature set; based on the distributed task big data feature set, perform task scheduling cluster analysis on the corresponding task computing nodes to generate task scheduling cluster groups corresponding to similar resource requirements; Step S3: Based on the task execution time and the actual usage of task resources, the remaining execution time of the task computing node corresponding to the distributed big data processing scenario is calculated to obtain the remaining execution calculation time of the task; based on the task deadline and the remaining execution calculation time of the task, the task urgency calculation is performed on the corresponding task computing node to obtain the scheduling urgency of the task node; Step S4: Determine the scheduling priority of the corresponding task computing node according to the scheduling urgency of the task node to obtain the task node scheduling priority; perform task scheduling optimization analysis on the task scheduling cluster group corresponding to similar resource requirements based on the task node scheduling priority to generate a task dynamic optimization scheduling plan.

[0006] Further, step S1 includes the following steps: Step S11: By configuring and deploying various distributed network devices between the server, storage device and client, a stable network connection between the devices in the corresponding task access test environment between the server, storage device and client is ensured, and a corresponding distributed big data processing scenario is generated; Step S12: collecting task-related data corresponding to the distributed access operation task in real time in the corresponding distributed big data processing scenario between the server, the storage device and the client, including task submission time, task deadline, task type and task resource requirements; Step S13: performing system resource usage calculation on the task computing nodes corresponding to the distributed big data processing scenario based on the task resource requirements to obtain task system resource usage data; Step S14: By deploying a data collection interface in the system kernel, task management module and resource monitoring tool location of the task computing node corresponding to the distributed big data processing scenario, and using the data collection interface to measure the task status of the task computing node corresponding to the distributed big data processing scenario, the task execution status data is obtained, which includes the task execution time, the actual usage of task resources and the task completion status.

[0007] Furthermore, the task resource requirements described in step S12 specifically include resource requirements corresponding to CPU, memory, storage, and network bandwidth.

[0008] Further, step S13 includes the following steps: Step S131: Calculate the CPU usage of the task computing nodes corresponding to the distributed big data processing scenario based on the resource requirements corresponding to the CPU to obtain the CPU usage corresponding to each computing node; Step S132: performing memory idle accounting on the task computing nodes corresponding to the distributed big data processing scenario based on the resource demand corresponding to the memory, so as to obtain the memory idle amount corresponding to each computing node; Step S133: performing storage read and write statistical analysis on the task computing nodes corresponding to the distributed big data processing scenario based on the resource requirements corresponding to the storage, so as to obtain the storage read and write speed corresponding to each computing node; Step S134: Calculate the bandwidth occupancy of the task computing nodes corresponding to the distributed big data processing scenario based on the resource demand corresponding to the network bandwidth, so as to obtain the network bandwidth occupancy corresponding to each computing node; Step S135: Merge the CPU usage, memory free space, storage read / write speed, and network bandwidth occupancy corresponding to each computing node to obtain task system resource usage data.

[0009] Further, step S2 includes the following steps: Step S21: Using the Hadoop distributed framework to perform big data multi-source heterogeneous storage on task-related data, task system resource usage data, and task execution status data, to obtain a distributed task big data storage set; Step S22: performing big data cleaning on the distributed task big data storage set to remove noise, duplicate values ​​and abnormal values ​​in the data set to obtain a distributed task big data cleaning set; Step S23: normalizing the distributed task big data cleaning set to unify task resources corresponding to different magnitudes to the same scale to obtain a distributed task big data normalized set; Step S24: performing big data feature mining analysis on the normalized set of distributed task big data, using a text classification method to analyze the distributed task description to determine the category to which the task belongs, and using data mining technology to extract resource demand pattern features corresponding to the distributed task to obtain a distributed task big data feature set; Step S25: Perform task scheduling cluster analysis on the corresponding task computing nodes based on the distributed task big data feature set to generate task scheduling cluster groups corresponding to similar resource requirements.

[0010] Further, step S3 includes the following steps: Step S31: performing resource usage fluctuation calculation on the task computing nodes corresponding to the distributed big data processing scenario based on the actual usage of the task resources to obtain the resource usage fluctuation rate of the task node; Step S32: Calculate the remaining execution time of the task computing node corresponding to the distributed big data processing scenario based on the task execution time and the task node resource usage fluctuation rate to obtain the task remaining execution calculation time; Step S33: Based on the task deadline and the remaining execution calculation time of the task, the task scheduling emergency calculation formula is used to perform task emergency calculation on the corresponding task calculation node to obtain the task node scheduling urgency.

[0011] Further, step S32 includes the following steps: Step S321: Calculate the corresponding task node resource usage variance and task node resource usage covariance through task node resource usage volatility statistics; Step S322: performing resource load balancing evaluation on the task computing nodes corresponding to the distributed big data processing scenario based on the task node resource usage variance and the task node resource usage covariance to obtain the task node resource load balancing degree; Step S323: predicting the remaining resource capacity of the task for the corresponding task computing node based on the task node resource load balancing degree to obtain the remaining resource capacity of the task node; Step S324: Based on the task execution time and the remaining resource capacity of the task node, the remaining execution time of the corresponding task computing node is estimated to obtain the remaining execution computing time of the task.

[0012] Furthermore, the task scheduling emergency calculation formula described in step S33 is specifically: ; In the formula, Schedule the urgency of the task nodes, Calculate the number of nodes for the task, For the The system resource usage of each task computing node, Calculate the maximum resource usage of the node for the task, is the task deadline, is the current time point, Calculate the remaining execution time for the task.

[0013] Further, step S4 includes the following steps: Step S41: determining the scheduling priority of the corresponding task computing node according to the scheduling urgency of the task node to obtain the scheduling priority of the task node; Step S42: scheduling and executing the task scheduling cluster groups corresponding to similar resource requirements in order based on the task node scheduling priorities, so as to obtain the task node group scheduling arrangement sequence corresponding to each task scheduling cluster group; Step S43: Optimize inter-group task scheduling according to the intra-group scheduling sequence of task nodes corresponding to each task scheduling cluster group, prioritize inter-group task scheduling and execute the scheduling of task nodes corresponding to the task scheduling cluster group to generate a dynamic task optimization scheduling solution.

[0014] Furthermore, the present invention also provides a task scheduling optimization analysis system based on big data, which is used to execute the task scheduling optimization analysis method based on big data as described above. The task scheduling optimization analysis system based on big data includes: The distributed task big data collection module is used to collect task-related data in real time in the corresponding distributed big data processing scenarios between the server, storage device and client, including task submission time, task deadline, task type and task resource requirements; based on the task resource requirements, the system resource usage of the task computing nodes corresponding to the distributed big data processing scenario is calculated to obtain the task system resource usage data; the task status of the task computing nodes corresponding to the distributed big data processing scenario is measured to obtain the task execution status data, including the task execution time, the actual usage of task resources and the task completion status; The task resource demand scheduling clustering module is used to use the Hadoop distributed framework to perform big data feature mining and analysis on task-related data, task system resource usage data, and task execution status data to obtain a distributed task big data feature set; based on the distributed task big data feature set, the corresponding task computing nodes are clustered for task scheduling, thereby generating task scheduling cluster groups with similar resource requirements; The task node scheduling emergency assessment module is used to calculate the remaining execution time of the task computing nodes corresponding to the distributed big data processing scenario based on the task execution time and the actual usage of task resources to obtain the remaining execution calculation time of the task; based on the task deadline and the remaining execution calculation time of the task, the corresponding task computing nodes are calculated for task urgency, thereby obtaining the scheduling urgency of the task node; The task scheduling optimization analysis module is used to determine the scheduling priority of the corresponding task computing nodes according to the scheduling urgency of the task nodes to obtain the task node scheduling priority; based on the task node scheduling priority, task scheduling optimization analysis is performed on the task scheduling cluster groups corresponding to similar resource requirements to generate a dynamic optimization scheduling plan for the task.

[0015] Beneficial effects of the present invention: 1. The task scheduling optimization and analysis method based on big data proposed in the present invention has the beneficial effect of collecting task-related data in distributed big data processing scenarios in real time, and the collected data includes task submission time, task deadline, task type, and task resource requirements, etc., which ensures that the start and end time of each task, the type of task, and the computing resources required for the task can be accurately grasped during the task execution process. The acquisition of these data not only helps the system to perform accurate task planning in the early stage of task execution, but also effectively tracks the progress of tasks, avoiding task delays or resource waste caused by insufficient or excessive resource allocation. Furthermore, through the combination of system resource usage calculation and task execution status measurement, the resource consumption of computing nodes can be evaluated in real time, and detailed task execution feedback can be provided, including task execution time, actual resource usage, and task completion status, which provides accurate dynamic data for subsequent task scheduling, helping the system to make effective resource adjustments during the task, thereby improving the efficiency of computing resource utilization and the timeliness of task completion. Secondly, by applying the Hadoop distributed framework to perform big data feature mining and analysis on task-related data, task system resource usage data, and task execution status data, the implicit features and rules in the task execution process can be identified. This analysis process enables the system to extract similar features between tasks, providing a more accurate reference for subsequent task scheduling. By performing cluster analysis on task scheduling, tasks with similar resource requirements can be grouped into the same scheduling cluster group, thereby achieving efficient resource allocation. For example, some tasks require a large amount of computing resources or memory, while other tasks have lower requirements for computing resources, thereby avoiding mixed scheduling of tasks with mismatched resource requirements, avoiding resource contention, inefficiency, or task delays, and significantly improving the computing performance and task completion efficiency of the entire distributed big data processing platform. Then, by calculating the remaining execution time of the task and the actual resource usage of the task, the progress and remaining time of the task can be accurately and dynamically monitored. The calculation of the remaining execution time helps to predict the completion time of each task, thereby providing a basis for task scheduling and resource allocation. By combining the task deadline and the remaining execution time, the scheduling urgency of each task node can be evaluated. This analysis provides a basis for the task scheduling system to sort the priorities of tasks, ensuring that urgent tasks can get computing resources first, avoiding timeouts or incomplete tasks due to improper resource scheduling, thereby improving the service quality of the entire distributed big data processing system.Finally, by calculating the scheduling priority of task nodes and dynamically optimizing the task scheduling cluster groups, tasks can be efficiently scheduled according to the urgency and resource requirements of the tasks. The core of this process is to determine the scheduling priority of task nodes so that the system can accurately allocate resources according to the actual needs of the tasks. For tasks with high urgency, the scheduling priority will be higher to ensure that these tasks can obtain the required computing resources in a timely manner, while tasks with fewer resources and lower urgency can be delayed if resources permit. This priority scheduling based on task urgency can effectively avoid resource waste or task delays caused by unreasonable task scheduling. At the same time, priority-based scheduling optimization can balance between different task groups to avoid a cluster group occupying too many resources, thereby causing resource shortages for other tasks. This dynamically optimized scheduling method can greatly improve the overall performance of the system, fully tap the value of data at task computing nodes, reduce the waste of computing resources, and ensure that tasks are completed on time, thereby improving the execution efficiency of tasks.

[0016] 2. The task scheduling optimization and analysis system based on big data proposed in the present invention is composed of a distributed task big data acquisition module, a task resource demand scheduling clustering module, a task node scheduling emergency assessment module and a task scheduling optimization and analysis module as a whole. It can realize any task scheduling optimization and analysis method based on big data described in the present invention, and is used to combine the operations between computer programs running on each module to realize the task scheduling optimization and analysis method based on big data. The internal structures of the system cooperate with each other, which can greatly reduce duplication of work and manpower investment, and can quickly and effectively provide a more accurate and efficient task scheduling optimization and analysis process based on big data, thereby simplifying the operation flow of the task scheduling optimization and analysis system based on big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments thereof made with reference to the following drawings: Figure 1 A schematic diagram of the steps of the task scheduling optimization analysis method based on big data of the present invention; Figure 2 for Figure 1 Detailed step flow diagram of step S1; Figure 3 for Figure 2 Detailed step flow chart of step S13 in FIG. DETAILED DESCRIPTION

[0018] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0020] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0021] To achieve this, please refer to Figure 1 to Figure 2 The present invention provides a task scheduling optimization analysis method based on big data, the method comprising the following steps: Step S1: by collecting task-related data in real time in the corresponding distributed big data processing scenarios between the server, the storage device and the client, including the task submission time, the task deadline, the task type and the task resource requirements; based on the task resource requirements, the system resource usage of the task computing nodes corresponding to the distributed big data processing scenarios is calculated to obtain the task system resource usage data; the task status of the task computing nodes corresponding to the distributed big data processing scenarios is measured to obtain the task execution status data, including the task execution time, the actual usage of the task resources and the task completion status; Step S2: Using the Hadoop distributed framework, perform big data feature mining and analysis on task-related data, task system resource usage data, and task execution status data to obtain a distributed task big data feature set; based on the distributed task big data feature set, perform task scheduling cluster analysis on the corresponding task computing nodes to generate task scheduling cluster groups corresponding to similar resource requirements; Step S3: Based on the task execution time and the actual usage of task resources, the remaining execution time of the task computing node corresponding to the distributed big data processing scenario is calculated to obtain the remaining execution calculation time of the task; based on the task deadline and the remaining execution calculation time of the task, the task urgency calculation is performed on the corresponding task computing node to obtain the scheduling urgency of the task node; Step S4: Determine the scheduling priority of the corresponding task computing node according to the scheduling urgency of the task node to obtain the task node scheduling priority; perform task scheduling optimization analysis on the task scheduling cluster group corresponding to similar resource requirements based on the task node scheduling priority to generate a task dynamic optimization scheduling plan.

[0022] In the embodiment of the present invention, please refer to Figure 1 FIG. 1 is a schematic diagram of the steps of the task scheduling optimization analysis method based on big data of the present invention. In this example, the task scheduling optimization analysis method based on big data includes the following steps: Step S1: by collecting task-related data in real time in the corresponding distributed big data processing scenarios between the server, the storage device and the client, including the task submission time, the task deadline, the task type and the task resource requirements; based on the task resource requirements, the system resource usage of the task computing nodes corresponding to the distributed big data processing scenarios is calculated to obtain the task system resource usage data; the task status of the task computing nodes corresponding to the distributed big data processing scenarios is measured to obtain the task execution status data, including the task execution time, the actual usage of the task resources and the task completion status; In the embodiment of the present invention, in the distributed big data processing scenario, the collection of task-related data is crucial. First, when a task is submitted, the system automatically records information such as the task submission time, the task deadline, the task type, and the task resource requirements. The task resource requirements include the requirements for resources such as CPU, memory, storage space, and bandwidth. The data is collected in real time between the server, storage device, and client through a distributed data collection system, and stored in a distributed database in a standardized format. For the resource usage calculation of the task computing node, the resource usage of each node is obtained through an interface with a cluster management tool (such as YARN or Mesos), including the resource usage corresponding to CPU, memory, storage, and network bandwidth, and the resource allocation of each computing node is calculated to generate task system resource usage data. In addition, the measurement of the task execution status is monitored through the system log, the task execution time is automatically recorded through the timestamp, and the actual usage of the task resources is tracked in real time by the system monitoring tool. The task completion status is classified by judging whether the task is successfully terminated or whether there is an exception, and the task status data is automatically updated to finally obtain the task execution status data.

[0023] Step S2: Using the Hadoop distributed framework, perform big data feature mining and analysis on task-related data, task system resource usage data, and task execution status data to obtain a distributed task big data feature set; based on the distributed task big data feature set, perform task scheduling cluster analysis on the corresponding task computing nodes to generate task scheduling cluster groups corresponding to similar resource requirements; In an embodiment of the present invention, the Hadoop distributed framework is used to perform big data feature mining and analysis on the collected task data. First, the task-related data, task system resource usage data and task execution status data are cleaned and preprocessed to remove outliers and redundant information. Then, the MapReduce model or Apache Spark is applied to perform parallel data processing, and machine learning algorithms such as clustering analysis and regression analysis are combined to extract key features such as time features, resource consumption features, and execution efficiency features of the tasks to form a distributed task big data feature set. When performing task scheduling cluster analysis, tasks are clustered based on features such as task resource requirements and execution time. Through the K-means clustering algorithm or the hierarchical clustering algorithm, task clusters with similar resource requirements and execution times are identified. Each task cluster corresponds to a specific task scheduling cluster group. These cluster groups will help optimize subsequent scheduling strategies and ultimately generate task scheduling cluster groups corresponding to similar resource requirements.

[0024] Step S3: Based on the task execution time and the actual usage of task resources, the remaining execution time of the task computing node corresponding to the distributed big data processing scenario is calculated to obtain the remaining execution calculation time of the task; based on the task deadline and the remaining execution calculation time of the task, the task urgency calculation is performed on the corresponding task computing node to obtain the scheduling urgency of the task node; In an embodiment of the present invention, the task execution time and the actual usage of task resources are key indicators for calculating the remaining execution time of the task. In this step, the executed time and the consumed resources are first obtained through the task execution status data, so as to calculate the current remaining execution time of each task. In specific implementation, the system will combine the task type, task resource usage pattern and execution time, and establish a regression model based on historical data and real-time data to predict the remaining execution time. The task deadline is known. Therefore, the remaining execution calculation time of the task can be obtained by subtracting the remaining execution time of the task from the task deadline. Then, the urgency of the task node is evaluated by using the difference between the remaining execution calculation time of the task and the task deadline. The task urgency calculation method is to perform weighted calculation on the remaining execution time of the task to evaluate whether the task needs to be scheduled first. Tasks with higher urgency will be marked as high priority tasks, and finally the scheduling urgency of the task node is obtained.

[0025] Step S4: Determine the scheduling priority of the corresponding task computing node according to the scheduling urgency of the task node to obtain the task node scheduling priority; perform task scheduling optimization analysis on the task scheduling cluster group corresponding to similar resource requirements based on the task node scheduling priority to generate a task dynamic optimization scheduling plan.

[0026] In an embodiment of the present invention, the task scheduling priority can be calculated for each task according to the scheduling urgency of the task node. The priority calculation takes into account the task deadline, the remaining execution time, the task resource requirements and the urgency of the task. First, by setting a rule, the system can automatically assign a preliminary priority level, and tasks with higher urgency are assigned high priority. Then, the system will use an optimization algorithm, such as a genetic algorithm or a simulated annealing algorithm, to perform optimization analysis on task scheduling cluster groups with similar resource requirements. The optimization analysis is based on factors such as the task resource requirements, task execution time, remaining resources, etc., combined with the idle resources of the computing node, to dynamically optimize the task scheduling plan. During the optimization process, tasks with higher task scheduling priorities will be preferentially scheduled to be executed on computing nodes with sufficient resources. The system will generate a dynamic optimization scheduling plan for the task, schedule the task to the most suitable computing node, ensure that the task can be completed within the specified time, and at the same time improve the utilization rate of computing resources, and finally generate a dynamic optimization scheduling plan for the task.

[0027] Further, step S1 includes the following steps: Step S11: By configuring and deploying various distributed network devices between the server, storage device and client, a stable network connection between the devices in the corresponding task access test environment between the server, storage device and client is ensured, and a corresponding distributed big data processing scenario is generated; Step S12: collecting task-related data corresponding to the distributed access operation task in real time in the corresponding distributed big data processing scenario between the server, the storage device and the client, including task submission time, task deadline, task type and task resource requirements; Step S13: performing system resource usage calculation on the task computing nodes corresponding to the distributed big data processing scenario based on the task resource requirements to obtain task system resource usage data; Step S14: By deploying a data collection interface in the system kernel, task management module and resource monitoring tool location of the task computing node corresponding to the distributed big data processing scenario, and using the data collection interface to measure the task status of the task computing node corresponding to the distributed big data processing scenario, the task execution status data is obtained, which includes the task execution time, the actual usage of task resources and the task completion status.

[0028] As an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 Detailed step flow diagram of step S1 in FIG. 1 , in this embodiment, step S1 includes the following steps: Step S11: By configuring and deploying various distributed network devices between the server, storage device and client, a stable network connection between the devices in the corresponding task access test environment between the server, storage device and client is ensured, and a corresponding distributed big data processing scenario is generated; In an embodiment of the present invention, during the creation of a distributed big data processing scenario, the network connection between the server, storage device and client is first configured and deployed to ensure that data can be exchanged between these devices stably and effectively. In a specific implementation, a physical network connection is established between different devices using a high-speed network switch and a router. At the same time, logical isolation is performed by configuring technologies such as a virtual local area network (VLAN) and a virtual private network (VPN) to ensure the security and reliability of data transmission. In order to ensure the smooth transmission of data flows in a big data processing environment, it is necessary to reasonably allocate bandwidth according to the task load and traffic requirements of each device, use a network load balancer (such as F5 or Cisco's load balancing device) to achieve optimal traffic distribution, deploy an efficient storage network (such as Fibre Channel SAN or iSCSI) between the server and the storage device to increase the speed of data access, and ensure that the data transmission between the client and the computing node is also stable and efficient. In this way, it is ensured that the task access of each device in the entire distributed environment can proceed smoothly, and finally the corresponding distributed big data processing scenario is configured and generated.

[0029] Step S12: collecting task-related data corresponding to the distributed access operation task in real time in the corresponding distributed big data processing scenario between the server, the storage device and the client, including task submission time, task deadline, task type and task resource requirements; In an embodiment of the present invention, after completing the deployment of the distributed big data processing environment, the task operation data is monitored and recorded in real time through the configured data collection system. In the specific implementation, a distributed log collection tool (such as ELKStack, Logstash) is used to deploy a log collector on each node, and the submission time, deadline, type and resource requirements of the task are recorded in detail. Each task will generate a unique task identifier when it is submitted, and the tasks are uniformly scheduled and tracked through a task management system (such as Apache Oozie, Airflow). The submission time and deadline of the task are recorded through automatic timestamps and transmitted to the data warehouse through the log collection tool. The type and resource requirements of the task are standardized by configuring a predefined template or interface, collected in real time and uploaded to the data center for storage. All collected task-related data will be provided to subsequent resource scheduling and optimization analysis modules, and finally the task-related data of the distributed access operation task is obtained, including the task submission time, task deadline, task type and task resource requirements.

[0030] Step S13: performing system resource usage calculation on the task computing nodes corresponding to the distributed big data processing scenario based on the task resource requirements to obtain task system resource usage data; In an embodiment of the present invention, after obtaining the resource requirement data of the task, the system resource usage of the task computing node is calculated for the resource requirement of each task (that is, the resource requirement corresponding to the CPU, memory, storage and network bandwidth). Specifically, first, the resource utilization of each task computing node is monitored by using a cluster resource management tool (such as Apache YARN or Kubernetes). These tools can obtain the usage of resources such as the node's CPU, memory, disk IO and network bandwidth in real time, and perform predictive calculations based on the resource requirements and priorities of the task through algorithms (such as resource scheduling algorithms and task priority sorting algorithms). Through these tools, the amount of resources required for each task during execution can be obtained, and then the task resource scheduling can be optimized. In this process, combined with a machine learning model, different types of task resource requirements can be learned and modeled to further improve the accuracy and efficiency of resource calculation. The results of the resource usage calculation will serve as the basis for subsequent task scheduling and task execution monitoring, including the CPU usage rate, memory idleness, storage read and write speed, and network bandwidth occupancy of each computing node, and finally the task system resource usage data is obtained.

[0031] Step S14: By deploying a data collection interface in the system kernel, task management module and resource monitoring tool location of the task computing node corresponding to the distributed big data processing scenario, and using the data collection interface to measure the task status of the task computing node corresponding to the distributed big data processing scenario, the task execution status data is obtained, which includes the task execution time, the actual usage of task resources and the task completion status.

[0032] In an embodiment of the present invention, by deploying a data collection interface on each task computing node, the specific operation is to deploy a resource monitoring module and a task status monitoring module on the computing node kernel using a system monitoring tool (such as Prometheus or Zabbix), and these modules collect key indicator data during the task execution process, such as task execution time, resource usage (such as real-time occupancy of CPU and memory), and task completion status, etc. When the task is executed, the monitoring module will record these data in real time and transmit them to a central database or data warehouse through a data collection interface for storage and analysis. In addition, by combining a task management module (such as Oozie or Airflow), the system can trigger automatic collection operations at the start and end of the task, record the complete life cycle data of the task, and judge and feedback the status of the task according to preset conditions. After completing the status judgment, the system will update the execution result (success or failure) of the task to the database. The monitoring tool displays the execution process and resource utilization of the task through a visual dashboard (such as Grafana), further optimizes the resource scheduling of the task, and finally obtains the task execution status data, specifically including the task execution time, the actual usage of task resources, and the task completion status.

[0033] Furthermore, the task resource requirements described in step S12 specifically include resource requirements corresponding to CPU, memory, storage, and network bandwidth.

[0034] Further, step S13 includes the following steps: Step S131: Calculate the CPU usage of the task computing nodes corresponding to the distributed big data processing scenario based on the resource requirements corresponding to the CPU to obtain the CPU usage corresponding to each computing node; Step S132: performing memory idle accounting on the task computing nodes corresponding to the distributed big data processing scenario based on the resource demand corresponding to the memory, so as to obtain the memory idle amount corresponding to each computing node; Step S133: performing storage read and write statistical analysis on the task computing nodes corresponding to the distributed big data processing scenario based on the resource requirements corresponding to the storage, so as to obtain the storage read and write speed corresponding to each computing node; Step S134: Calculate the bandwidth occupancy of the task computing nodes corresponding to the distributed big data processing scenario based on the resource demand corresponding to the network bandwidth, so as to obtain the network bandwidth occupancy corresponding to each computing node; Step S135: Merge the CPU usage, memory free space, storage read / write speed, and network bandwidth occupancy corresponding to each computing node to obtain task system resource usage data.

[0035] As an embodiment of the present invention, refer to Figure 3 As shown, Figure 2 FIG. 1 is a schematic diagram of a detailed step flow chart of step S13 in FIG. 1 . In this embodiment, step S13 includes the following steps: Step S131: Calculate the CPU usage of the task computing nodes corresponding to the distributed big data processing scenario based on the resource requirements corresponding to the CPU to obtain the CPU usage corresponding to each computing node; In an embodiment of the present invention, by obtaining the CPU resource information of each computing node, including the total number of CPU cores, frequency and the amount of used CPU resources of each computing node, monitoring tools such as top, htop, psutil, etc. can be used to obtain the current usage of each core on the computing node. Then, the CPU usage of the computing node can be obtained by real-time monitoring of the CPU load. For example, the CPU usage can be calculated by the following formula: CPU usage = used CPU time / total CPU time × 100%. In this step, the usage of each CPU core of each computing node is statistically summarized to finally obtain the CPU usage corresponding to each computing node.

[0036] Step S132: performing memory idle accounting on the task computing nodes corresponding to the distributed big data processing scenario based on the resource demand corresponding to the memory, so as to obtain the memory idle amount corresponding to each computing node; In an embodiment of the present invention, by obtaining the total capacity of the memory on each computing node and the currently used memory capacity, the total amount and remaining amount of memory can be extracted by using, for example, the free command, the vmstat command or the psutil library. Specifically, the total amount of memory on each computing node is first determined, and the amount of used memory is obtained. Then, the amount of free memory is calculated by the following formula: free memory = total memory - used memory. The free memory amount will reflect whether each computing node has sufficient memory to process new tasks. This calculation can not only help evaluate the current load of the computing node, but also provide a basis for optimizing the utilization of memory resources, and finally obtain the free memory amount corresponding to each computing node.

[0037] Step S133: performing storage read and write statistical analysis on the task computing nodes corresponding to the distributed big data processing scenario based on the resource requirements corresponding to the storage, so as to obtain the storage read and write speed corresponding to each computing node; In an embodiment of the present invention, by obtaining the read and write rate information of the computing node storage device, the I / O operation status of the disk can be obtained through monitoring tools such as iostat, dstat, etc. In order to obtain the accurate storage read and write speed, the following steps can be performed: first read the total disk I / O read and write bytes of each computing node, and then calculate the storage read and write speed according to the set time window, for example: storage read and write speed = total read and write bytes / time window. This will provide performance data of the storage device for each computing node and help evaluate whether there is resource consumption or delay caused by the disk I / O bottleneck, and finally obtain the storage read and write speed corresponding to each computing node.

[0038] Step S134: Calculate the bandwidth occupancy of the task computing nodes corresponding to the distributed big data processing scenario based on the resource demand corresponding to the network bandwidth, so as to obtain the network bandwidth occupancy corresponding to each computing node; In an embodiment of the present invention, by monitoring the network bandwidth usage of each computing node, the network bandwidth occupancy is usually monitored and recorded in real time by tools such as iftop, netstat, and nload. By obtaining the upload and download traffic data of the network interface of each computing node, the network bandwidth usage of each node is calculated. The calculation method is: bandwidth occupancy = data transmission volume / time window. Through this method, the bandwidth consumption of each computing node during data transmission can be evaluated, and it can be understood whether the network becomes a potential bottleneck, and finally the network bandwidth occupancy corresponding to each computing node can be obtained.

[0039] Step S135: Merge the CPU usage, memory free space, storage read / write speed, and network bandwidth occupancy corresponding to each computing node to obtain task system resource usage data.

[0040] In an embodiment of the present invention, by integrating the various resource indicators (CPU usage, memory idle amount, storage read and write speed, network bandwidth occupancy) calculated in steps S131 to S134, the specific operation can integrate the corresponding CPU usage, memory idle amount, storage read and write speed, and network bandwidth occupancy into the same big data set, and finally obtain the task system resource usage data.

[0041] Further, step S2 includes the following steps: Step S21: Using the Hadoop distributed framework to perform big data multi-source heterogeneous storage on task-related data, task system resource usage data, and task execution status data, to obtain a distributed task big data storage set; In the embodiment of the present invention, the Hadoop distributed framework is used to perform multi-source heterogeneous storage on task-related data, task system resource usage data, and task execution status data. HDFS (Hadoop Distributed File System) is used as the storage layer to ensure that all data can be distributedly stored among multiple nodes. The task-related data includes task submission time, task deadline, task type, and task resource requirements; the task system resource usage data includes the CPU, memory, disk I / O (i.e., storage), and network bandwidth usage requirements of each task; the task execution status data records the task execution time, the actual usage of task resources, and the task completion status (success, failure, or suspension), etc. The Hadoop MapReduce framework is used for preliminary data processing and distribution. The task data is stored as different data files in HDFS. By setting a reasonable sharding strategy, large-scale task data is partitioned and stored to improve subsequent processing efficiency, ensure data reliability and availability, and finally form a distributed task big data storage set.

[0042] Step S22: performing big data cleaning on the distributed task big data storage set to remove noise, duplicate values ​​and abnormal values ​​in the data set to obtain a distributed task big data cleaning set; In an embodiment of the present invention, by performing data cleaning on a distributed task big data storage set, Apache Spark is first used as a data processing engine for parallel computing. In the data cleaning stage, a deduplication algorithm, such as hash deduplication, timestamp-based deduplication, etc., is used to clean up duplicate task data. Then, statistical methods such as Z-Score or IQR (Interquartile Range) are used to detect and process outliers, identify abnormal data in task execution time and resource usage, and make corresponding corrections or deletions. For noise data, multiple data filtering mechanisms can be constructed to eliminate records that do not conform to the expected pattern, such as using the K-Means clustering method to identify noise points, and smoothing the data through the weighted average method. After cleaning, a distributed task big data cleaning set is finally obtained.

[0043] Step S23: normalizing the distributed task big data cleaning set to unify task resources corresponding to different magnitudes to the same scale to obtain a distributed task big data normalized set; In an embodiment of the present invention, data normalization is performed on the cleaned distributed task big data cleaning set, with the goal of converting resource items with large magnitude differences in different task data (such as CPU usage, memory usage, disk I / O, network bandwidth, etc.) to the same scale, and using the minimum-maximum normalization method (Min-Max Normalization) to process all task resources, and scale the resource indicators of each task to between 0 and 1 to eliminate the impact of different resource usage indicator dimensions, ensuring that various indicators are compared at the same scale. Through the standardized data, the deviation caused by different data units can be eliminated, so that each task has a relatively fair weight in the resource scheduling process. In addition, the Z-Score standardization method can be used to standardize the data with a mean of 0 and a variance of 1 to adapt to different machine learning and data mining algorithms, and finally obtain a normalized set of distributed task big data.

[0044] Step S24: performing big data feature mining analysis on the normalized set of distributed task big data, using a text classification method to analyze the distributed task description to determine the category to which the task belongs, and using data mining technology to extract resource demand pattern features corresponding to the distributed task to obtain a distributed task big data feature set; In an embodiment of the present invention, after data normalization, feature mining analysis is performed on the normalized set of distributed task big data to analyze the description of the task by using a text classification method, and with the help of natural language processing (NLP) technology, the TF-IDF (term frequency-inverse document frequency) algorithm is used to extract keywords in the task description, and the task category is identified by a naive Bayes classifier or a support vector machine (SVM) model to determine the category to which each task belongs. Data mining technology is further applied to extract resource demand pattern features corresponding to each task from the cleaned task execution data, and through a clustering analysis algorithm, such as K-Means or DBSCAN, combined with the resource consumption behavior during the task execution process, the demand pattern of the task for computing resources is automatically identified. These pattern features will help understand the resource demand law of the task, thereby providing a basis for subsequent scheduling and resource allocation, and finally a distributed task big data feature set is obtained, including key information such as the task category and resource demand features.

[0045] Step S25: Perform task scheduling cluster analysis on the corresponding task computing nodes based on the distributed task big data feature set to generate task scheduling cluster groups corresponding to similar resource requirements.

[0046] In an embodiment of the present invention, based on the distributed task big data feature set obtained above, task scheduling clustering analysis is further performed on the task computing nodes. First, the K-Means algorithm is used to cluster the tasks. The goal of clustering is to divide tasks with similar resource demand patterns into a cluster. The tasks of each cluster have high similarity in the demand for computing resources when executed, and can be scheduled on the same or similar computing nodes to avoid resource waste and excessive competition. By calculating the similarity of resource requirements between tasks, the clustering algorithm can automatically generate multiple task scheduling cluster groups. Tasks in each cluster group can share resources, thereby improving the resource utilization of the system. In addition, clustering analysis can also help identify the dependencies between tasks, and provide strong support for optimizing the execution order and scheduling strategy of tasks. Through task scheduling clustering analysis, task scheduling cluster groups with similar resource requirements are finally generated to promote efficient scheduling and execution of tasks.

[0047] Further, step S3 includes the following steps: Step S31: performing resource usage fluctuation calculation on the task computing nodes corresponding to the distributed big data processing scenario based on the actual usage of the task resources to obtain the resource usage fluctuation rate of the task node; In an embodiment of the present invention, by collecting indicators such as CPU usage, memory consumption, storage and network bandwidth usage of task nodes within a certain period of time, a resource usage model is established based on historical data. This model calculates the resource usage fluctuation of the node at each moment, and quantifies the volatility through statistical methods such as standard deviation and coefficient of variation. By real-time monitoring of the resource consumption of each computing node, the resource usage volatility of each node is obtained, reflecting the amplitude of the node resource consumption fluctuation. In order to accurately calculate the volatility, the system needs to discretize the resource usage data of the node in each time period, and further eliminate extreme fluctuations through methods such as sliding windows or weighted averages, and finally obtain the resource usage volatility of the task node.

[0048] Step S32: Calculate the remaining execution time of the task computing node corresponding to the distributed big data processing scenario based on the task execution time and the task node resource usage fluctuation rate to obtain the task remaining execution calculation time; In an embodiment of the present invention, after obtaining the resource usage volatility of the task computing node, the system further calculates the remaining execution time of the task in combination with the actual execution time of the task and the fluctuation of the node. First, the system needs to obtain the current execution progress of the task and the time consumed. This information is usually tracked in real time by the monitoring module in the task scheduling system. On this basis, the system estimates the resource volatility of the task node. If the volatility of a certain node is large, it will lead to instability in the calculation progress, thereby extending the execution time of the task. By comprehensively considering the impact of the volatility, the system uses linear regression or multivariate prediction models to correct the prediction of the remaining execution time. In this process, it is necessary to take into account factors such as the node's computing load and network delay, and adjust the model according to the characteristics of the task (such as computing intensive or IO intensive), and finally obtain the remaining execution calculation time of the task.

[0049] Step S33: Based on the task deadline and the remaining execution calculation time of the task, the task scheduling emergency calculation formula is used to perform task emergency calculation on the corresponding task calculation node to obtain the task node scheduling urgency.

[0050] In an embodiment of the present invention, a suitable task scheduling emergency calculation formula is formed by combining the number of task computing nodes, system resource usage, maximum resource usage of task computing nodes, task deadline, current time point, remaining execution calculation time of the task and related parameters to perform quantitative calculation of the task urgency, so as to quantitatively calculate the remaining available time window of the task. An emergency coefficient formula may also be used: urgency = (task deadline - current time) / remaining execution time. In this formula, if the urgency is close to 1, it indicates that the task is very urgent and must be processed immediately; if the urgency is lower, the task can be scheduled in a subsequent time period, and finally the task node scheduling urgency is obtained.

[0051] Further, step S32 includes the following steps: Step S321: Calculate the corresponding task node resource usage variance and task node resource usage covariance through task node resource usage volatility statistics; In an embodiment of the present invention, by collecting and counting the resource usage of each task node, these resources include but are not limited to CPU utilization, memory usage, network bandwidth, etc., in order to evaluate the volatility of task node resource usage, it is necessary to perform statistical analysis on the resource consumption data of each node within a certain period of time, calculate its volatility, and calculate the volatility based on the standard deviation and average value of resource usage, and then obtain the variance. Then, for the resource usage data between different task nodes, the covariance is calculated to understand the correlation between the resource usage of each task node. This process relies on the time series analysis method, and uses statistical tools such as MATLAB, R language or the pandas library in Python to complete the calculation of variance and covariance. In specific implementation, first, the data set of each task node is cleaned to remove outliers, and then the variance is calculated using the sum of squares of the deviation of the data, and then the covariance of the task node resource usage is obtained through the covariance formula, and finally the variance of the task node resource usage and the covariance of the task node resource usage are obtained.

[0052] Step S322: performing resource load balancing evaluation on the task computing nodes corresponding to the distributed big data processing scenario based on the task node resource usage variance and the task node resource usage covariance to obtain the task node resource load balancing degree; In an embodiment of the present invention, the previously obtained variance and covariance of resource usage of task nodes can provide key data for load balancing evaluation. During specific operations, the stability of node resource usage is first judged based on the resource variance of the task node. A larger variance indicates that the usage of the node resources fluctuates greatly and the load pressure may be greater; conversely, a smaller variance indicates that the resource usage is relatively stable. Then, the covariance is used to evaluate the resource usage dependency between the task nodes. If the covariance is high, it indicates that there is a strong correlation between the resource usage changes of the two nodes and coordinated scheduling is required. The load balancing degree can be quantified by comprehensively considering the variance and covariance between the nodes. Generally speaking, the goal of load balancing is to make the resource usage of each task node more uniform by adjusting the task allocation. During evaluation, a weighted average method or a linear programming method is used to comprehensively evaluate each node to obtain the resource load balancing degree. The balance degree measures the uniformity of the load of each computing node in the system, and finally the task node resource load balancing degree is obtained.

[0053] Step S323: predicting the remaining resource capacity of the task for the corresponding task computing node based on the task node resource load balancing degree to obtain the remaining resource capacity of the task node; In an embodiment of the present invention, the remaining resource capacity of each node is evaluated based on the previously obtained task node resource load balance. A node with a lower load balance indicates that its resources are relatively overloaded, so it is necessary to predict its remaining resources through scheduling optimization. In order to calculate the remaining resource capacity of each task node, the current resource occupancy of the node can be predicted, and the resource usage trend can be predicted by using a time series prediction method (such as an ARIMA model or an LSTM neural network) in combination with the historical resource usage data of the task node. The input of the prediction model is the historical resource usage data of the node, and the output is the predicted resource usage value of the node in the future time period. The remaining resource capacity of each node is obtained by comparing the difference between the total resource capacity of the node and the predicted resource demand. For different types of tasks (such as compute-intensive tasks or IO-intensive tasks), different prediction algorithms can be used to ensure the accuracy of the prediction, and finally the remaining resource capacity of the task node is obtained.

[0054] Step S324: Based on the task execution time and the remaining resource capacity of the task node, the remaining execution time of the corresponding task computing node is estimated to obtain the remaining execution computing time of the task.

[0055] In an embodiment of the present invention, the estimation of the remaining execution time needs to be performed based on the execution time of the task and the remaining resource capacity of the task node. Specifically, the execution progress information of the current task is first collected, including the completed task volume and the estimated total execution time. Combined with the previously obtained remaining resource capacity of the task node, the relationship model between the task execution time and resource consumption is used to estimate the execution time. Usually, by analyzing the resource consumption and progress of the task when running on different nodes, a resource-time mapping model can be established. The model can estimate the remaining execution time based on the remaining resources of the node. By comparing the current progress of the task with the expected completion progress, combined with the resource consumption of the current node, the remaining execution time can be predicted. In order to improve the estimation accuracy, regression analysis and machine learning algorithms (such as decision trees or support vector machines) can be used to train and optimize the mapping relationship between task execution time and resource capacity. This method takes into account the task type, resource usage pattern and historical execution data, thereby improving the estimation accuracy of the remaining execution time of the task, and finally obtaining the remaining execution calculation time of the task.

[0056] Furthermore, the task scheduling emergency calculation formula described in step S33 is specifically: ; In the formula, Schedule urgency for task nodes, Calculate the number of nodes for the task, For the The system resource usage of each task computing node, Calculate the maximum resource usage of the node for the task, is the task deadline, is the current time point, Calculate the remaining execution time for the task.

[0057] The present invention obtains a task scheduling emergency calculation formula by using a specific mathematical model and verifying it, which is used to perform task emergency calculations on the corresponding task computing nodes. The task scheduling emergency calculation formula realizes dynamic prediction of task execution time by taking into account the resource usage fluctuations of the task computing nodes, which means that task scheduling can adapt to the current resource usage and avoid computing node overload or uneven resource distribution. The formula takes into account the resource usage and maximum resource usage of the task nodes, and accurately reflects the changes in system resource requirements of the task nodes in different time periods, thereby providing a real-time reference for task scheduling decisions. Fluctuations in the resource usage of the computing nodes (such as CPU, memory, storage, etc.) during the execution process will affect the remaining execution time of the task. Secondly, the formula The remaining time of the task is combined with the deadline to reflect the urgency of task completion. In this way, the scheduling system can better evaluate the matching degree between the remaining time of the current task and the deadline, so as to reasonably arrange the task allocation of the computing nodes to ensure that the task is completed on time. In summary, this formula fully considers the urgency of task node scheduling. , the number of task computing nodes , No. System resource usage of each task computing node , the maximum resource usage of the task computing node , task deadline , current time point , the remaining execution time of the task , according to the urgency of task node scheduling The correlation between the above parameters constitutes a functional relationship This formula can realize the task emergency calculation process of the corresponding task calculation node, thereby improving the accuracy and applicability of the task scheduling emergency calculation formula.

[0058] Further, step S4 includes the following steps: Step S41: determining the scheduling priority of the corresponding task computing node according to the scheduling urgency of the task node to obtain the scheduling priority of the task node; In an embodiment of the present invention, a numerical urgency level is assigned to each task node based on the scheduling urgency of the task nodes obtained through previous quantitative calculations. Task nodes with high scheduling urgency (such as tasks approaching deadlines or requiring a large amount of computing resources) will be given a higher priority. Afterwards, the scheduling system sorts the task nodes according to their urgency, and obtains the scheduling priority of each task node through algorithms such as weighted sorting or priority queue method. In this process, if the urgency of multiple task nodes is the same, the sorting is further refined according to the computational complexity or dependency relationship of the task nodes to ensure the efficiency and accuracy of task scheduling, and finally obtain the scheduling priority of the task node.

[0059] Step S42: scheduling and executing the task scheduling cluster groups corresponding to similar resource requirements in order based on the task node scheduling priorities, so as to obtain the task node group scheduling arrangement sequence corresponding to each task scheduling cluster group; In an embodiment of the present invention, by analyzing the resource requirements of task nodes, task nodes with similar resource requirements are identified, and these task nodes are assigned to different scheduling cluster groups. The task nodes in each task scheduling cluster group have similar resource requirement characteristics, such as the number of CPU cores, memory size, storage capacity, etc. When arranging the scheduling execution order of these task nodes in the same cluster group, the scheduling priority of the task nodes must be given priority. For the task nodes in the same cluster group, the scheduling system sorts them according to the previously determined task node priorities, and prioritizes the scheduling of task nodes with high urgency. If there are multiple task nodes with the same priority in the cluster group, the scheduling system further uses other scheduling strategies, such as first come first served (FCFS) or shortest job first (SJF), etc., to finely arrange the execution order of the task nodes to ensure efficient utilization of resources and balanced task execution, and finally obtain the scheduling arrangement sequence within the task node group corresponding to each task scheduling cluster group.

[0060] Step S43: Optimize inter-group task scheduling according to the intra-group scheduling sequence of task nodes corresponding to each task scheduling cluster group, prioritize inter-group task scheduling and execute the scheduling of task nodes corresponding to the task scheduling cluster group to generate a dynamic task optimization scheduling solution.

[0061] In an embodiment of the present invention, by obtaining the intra-group scheduling arrangement sequence of the task nodes of each task scheduling cluster group, the scheduling system performs inter-group task scheduling optimization based on these intra-group sorting results. In the inter-group optimization, the scheduling system will perform comprehensive scheduling of the task nodes of different cluster groups to ensure that when multiple tasks are executed concurrently, the task scheduling order between different cluster groups can balance the use of system resources. Cluster groups with higher scheduling priorities will be given priority support in resource allocation, while cluster groups with lower priorities will be scheduled according to idle resources. When the task nodes of a cluster group are executed, the task nodes of the next cluster group will be immediately scheduled for execution according to dynamic resource availability to avoid resource waste. The scheduling system generates a dynamic optimization scheduling plan based on the above optimization results. The plan can achieve optimal resource allocation, improve task execution efficiency, and ensure that high-priority tasks can be completed on time, and finally generate a task dynamic optimization scheduling plan.

[0062] Furthermore, the present invention also provides a task scheduling optimization analysis system based on big data, which is used to execute the task scheduling optimization analysis method based on big data as described above. The task scheduling optimization analysis system based on big data includes: The distributed task big data collection module is used to collect task-related data in real time in the corresponding distributed big data processing scenarios between the server, storage device and client, including task submission time, task deadline, task type and task resource requirements; based on the task resource requirements, the system resource usage of the task computing nodes corresponding to the distributed big data processing scenario is calculated to obtain the task system resource usage data; the task status of the task computing nodes corresponding to the distributed big data processing scenario is measured to obtain the task execution status data, including the task execution time, the actual usage of task resources and the task completion status; The task resource demand scheduling clustering module is used to use the Hadoop distributed framework to perform big data feature mining and analysis on task-related data, task system resource usage data, and task execution status data to obtain a distributed task big data feature set; based on the distributed task big data feature set, the corresponding task computing nodes are clustered for task scheduling, thereby generating task scheduling cluster groups with similar resource requirements; The task node scheduling emergency assessment module is used to calculate the remaining execution time of the task computing nodes corresponding to the distributed big data processing scenario based on the task execution time and the actual usage of task resources to obtain the remaining execution calculation time of the task; based on the task deadline and the remaining execution calculation time of the task, the corresponding task computing nodes are calculated for task urgency, thereby obtaining the scheduling urgency of the task node; The task scheduling optimization analysis module is used to determine the scheduling priority of the corresponding task computing nodes according to the scheduling urgency of the task nodes to obtain the task node scheduling priority; based on the task node scheduling priority, task scheduling optimization analysis is performed on the task scheduling cluster groups corresponding to similar resource requirements to generate a dynamic optimization scheduling plan for the task.

[0063] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0064] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A task scheduling optimization analysis method based on big data, characterized in that: The following steps are involved: Step S1: collecting task-related data in real time in the corresponding distributed big data processing scenarios between the server, storage device and client, including task submission time, task deadline, task type and task resource requirements; Based on the task resource requirements, the system resource usage of the task computing nodes corresponding to the distributed big data processing scenario is calculated to obtain the task system resource usage data; the task status of the task computing nodes corresponding to the distributed big data processing scenario is measured to obtain the task execution status data, including the task execution time, the actual usage of task resources and the task completion status; Step S2: Using the Hadoop distributed framework, perform big data feature mining and analysis on task-related data, task system resource usage data, and task execution status data to obtain a distributed task big data feature set; based on the distributed task big data feature set, perform task scheduling cluster analysis on the corresponding task computing nodes to generate task scheduling cluster groups corresponding to similar resource requirements; Step S3: Calculate the remaining execution time of the task computing node corresponding to the distributed big data processing scenario based on the task execution time and the actual usage of the task resources to obtain the remaining execution calculation time of the task; Based on the task deadline and the remaining execution calculation time of the task, the corresponding task computing node is calculated for task urgency, and the scheduling urgency of the task node is obtained; Step S4: Determine the scheduling priority of the corresponding task computing node according to the scheduling urgency of the task node to obtain the scheduling priority of the task node; Based on the scheduling priority of task nodes, task scheduling optimization analysis is performed on task scheduling cluster groups corresponding to similar resource requirements to generate a dynamic optimization scheduling solution for tasks.

2. The task scheduling optimization analysis method based on big data according to claim 1 is characterized in that: Step S1 includes the following steps: Step S11: By configuring and deploying various distributed network devices between the server, storage device and client, a stable network connection between the devices in the corresponding task access test environment between the server, storage device and client is ensured, and a corresponding distributed big data processing scenario is generated; Step S12: collecting task-related data corresponding to the distributed access operation task in real time in the corresponding distributed big data processing scenario between the server, the storage device and the client, including task submission time, task deadline, task type and task resource requirements; Step S13: performing system resource usage calculation on the task computing nodes corresponding to the distributed big data processing scenario based on the task resource requirements to obtain task system resource usage data; Step S14: By deploying a data collection interface in the system kernel, task management module and resource monitoring tool location of the task computing node corresponding to the distributed big data processing scenario, and using the data collection interface to measure the task status of the task computing node corresponding to the distributed big data processing scenario, the task execution status data is obtained, which includes the task execution time, the actual usage of task resources and the task completion status.

3. The task scheduling optimization analysis method based on big data according to claim 2 is characterized in that: The task resource requirements described in step S12 specifically include resource requirements corresponding to CPU, memory, storage, and network bandwidth.

4. The task scheduling optimization analysis method based on big data according to claim 3 is characterized in that: Step S13 includes the following steps: Step S131: Calculate the CPU usage of the task computing nodes corresponding to the distributed big data processing scenario based on the resource requirements corresponding to the CPU to obtain the CPU usage corresponding to each computing node; Step S132: performing memory idle accounting on the task computing nodes corresponding to the distributed big data processing scenario based on the resource demand corresponding to the memory, so as to obtain the memory idle amount corresponding to each computing node; Step S133: performing storage read and write statistical analysis on the task computing nodes corresponding to the distributed big data processing scenario based on the resource requirements corresponding to the storage, so as to obtain the storage read and write speed corresponding to each computing node; Step S134: Calculate the bandwidth occupancy of the task computing nodes corresponding to the distributed big data processing scenario based on the resource demand corresponding to the network bandwidth, so as to obtain the network bandwidth occupancy corresponding to each computing node; Step S135: Merge the CPU usage, memory free space, storage read / write speed, and network bandwidth occupancy corresponding to each computing node to obtain task system resource usage data.

5. The task scheduling optimization analysis method based on big data according to claim 1 is characterized in that: Step S2 includes the following steps: Step S21: Using the Hadoop distributed framework to perform big data multi-source heterogeneous storage on task-related data, task system resource usage data, and task execution status data, to obtain a distributed task big data storage set; Step S22: performing big data cleaning on the distributed task big data storage set to remove noise, duplicate values ​​and abnormal values ​​in the data set to obtain a distributed task big data cleaning set; Step S23: normalizing the distributed task big data cleaning set to unify task resources corresponding to different magnitudes to the same scale to obtain a distributed task big data normalized set; Step S24: performing big data feature mining analysis on the normalized set of distributed task big data, using a text classification method to analyze the distributed task description to determine the category to which the task belongs, and using data mining technology to extract resource demand pattern features corresponding to the distributed task to obtain a distributed task big data feature set; Step S25: Perform task scheduling cluster analysis on the corresponding task computing nodes based on the distributed task big data feature set to generate task scheduling cluster groups corresponding to similar resource requirements.

6. The task scheduling optimization analysis method based on big data according to claim 1 is characterized in that: Step S3 includes the following steps: Step S31: performing resource usage fluctuation calculation on the task computing nodes corresponding to the distributed big data processing scenario based on the actual usage of the task resources to obtain the resource usage fluctuation rate of the task node; Step S32: Calculate the remaining execution time of the task computing node corresponding to the distributed big data processing scenario based on the task execution time and the task node resource usage fluctuation rate to obtain the task remaining execution calculation time; Step S33: Based on the task deadline and the remaining execution calculation time of the task, the task scheduling emergency calculation formula is used to perform task emergency calculation on the corresponding task calculation node to obtain the task node scheduling urgency.

7. The task scheduling optimization analysis method based on big data according to claim 6 is characterized in that: Step S32 includes the following steps: Step S321: Calculate the corresponding task node resource usage variance and task node resource usage covariance through task node resource usage volatility statistics; Step S322: performing resource load balancing evaluation on the task computing nodes corresponding to the distributed big data processing scenario based on the task node resource usage variance and the task node resource usage covariance to obtain the task node resource load balancing degree; Step S323: predicting the remaining resource capacity of the task for the corresponding task computing node based on the task node resource load balancing degree to obtain the remaining resource capacity of the task node; Step S324: Based on the task execution time and the remaining resource capacity of the task node, the remaining execution time of the corresponding task computing node is estimated to obtain the remaining execution computing time of the task.

8. The task scheduling optimization analysis method based on big data according to claim 6 is characterized in that: The task scheduling emergency calculation formula described in step S33 is specifically: ； In the formula, Schedule urgency for task nodes, Calculate the number of nodes for the task, For the The system resource usage of each task computing node, Calculate the maximum resource usage of the node for the task, is the task deadline, is the current time point, Calculate the remaining execution time for the task.

9. The task scheduling optimization analysis method based on big data according to claim 1 is characterized in that: Step S4 includes the following steps: Step S41: determining the scheduling priority of the corresponding task computing node according to the scheduling urgency of the task node to obtain the scheduling priority of the task node; Step S42: scheduling and executing the task scheduling cluster groups corresponding to similar resource requirements in order based on the task node scheduling priorities, so as to obtain the task node group scheduling arrangement sequence corresponding to each task scheduling cluster group; Step S43: Optimize inter-group task scheduling according to the intra-group scheduling sequence of task nodes corresponding to each task scheduling cluster group, prioritize inter-group task scheduling and execute the scheduling of task nodes corresponding to the task scheduling cluster group to generate a dynamic task optimization scheduling solution.

10. A task scheduling optimization and analysis system based on big data, characterized in that: Used to execute the task scheduling optimization analysis method based on big data as claimed in claim 1, the task scheduling optimization analysis system based on big data comprises: The distributed task big data collection module is used to collect task-related data in real time in the corresponding distributed big data processing scenarios between the server, storage device and client, including task submission time, task deadline, task type and task resource requirements; based on the task resource requirements, the system resource usage of the task computing nodes corresponding to the distributed big data processing scenario is calculated to obtain the task system resource usage data; the task status of the task computing nodes corresponding to the distributed big data processing scenario is measured to obtain the task execution status data, including the task execution time, the actual usage of task resources and the task completion status; The task resource demand scheduling clustering module is used to use the Hadoop distributed framework to perform big data feature mining and analysis on task-related data, task system resource usage data, and task execution status data to obtain a distributed task big data feature set; based on the distributed task big data feature set, the corresponding task computing nodes are clustered for task scheduling, thereby generating task scheduling cluster groups with similar resource requirements; The task node scheduling emergency assessment module is used to calculate the remaining execution time of the task computing nodes corresponding to the distributed big data processing scenario based on the task execution time and the actual usage of task resources to obtain the remaining execution calculation time of the task; based on the task deadline and the remaining execution calculation time of the task, the corresponding task computing nodes are calculated for task urgency, thereby obtaining the scheduling urgency of the task node; The task scheduling optimization analysis module is used to determine the scheduling priority of the corresponding task computing nodes according to the scheduling urgency of the task nodes to obtain the task node scheduling priority; based on the task node scheduling priority, task scheduling optimization analysis is performed on the task scheduling cluster groups corresponding to similar resource requirements to generate a dynamic optimization scheduling plan for the task.

Images