Resource dynamic scheduling method for big data platform computing tasks

Abstract

The invention relates to the technical field of big data and cloud computing, in particular to a resource dynamic scheduling method for big data platform computing tasks. The method comprises the steps of receiving submitted calculation tasks and task resource demand declarations, collecting state data of a plurality of servers in a resource pool, sorting and matching task queues, sending occupation instructions to the matched servers, and distributing corresponding calculation tasks to the servers. The method comprises the following steps: monitoring a real-time execution state of a server, completing addition of the server in a resource pool, cleaning task calculation resources after a calculation task is completed, returning the task calculation resources to the resource pool, judging resource dynamic scheduling of the big data platform calculation task based on task completion duration, judging whether a standard is met or not based on core dependent resources, and if yes, executing the calculation task. And adjustment is carried out according to corresponding parameters. According to the method, accurate resource dynamic scheduling of the big data platform computing tasks is effectively realized, and the resource dynamic scheduling efficiency of the big data platform computing tasks is effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of big data and cloud computing technology, and in particular to a method for dynamic resource scheduling of computing tasks on a big data platform. Background Technology

[0002] Traditional data processing tools have fixed server numbers, storage capacity, and computing power at deployment. This makes it difficult to quickly adjust server resources based on business fluctuations, easily leading to resource waste or shortages. This limitation is inherent to the architecture of traditional data processing tools and cannot be solved through simple optimization. Big data platforms, on the other hand, possess a flexible management architecture, with dynamic scheduling as one of their core features. Big data platforms connect to poolable server resources, integrating the hardware capabilities of multiple servers into a unified, resource-allocated pool. Servers no longer operate independently but are packaged into a whole through the technology of the big data platform, allowing for flexible access or redeployment based on task requirements. Each server in the resource pool is an independent hardware device, potentially located in the same data center or in different regions. With the development of technology, research on the dynamic scheduling of computing tasks on big data platforms is of great significance.

[0003] Chinese Patent Publication No. CN116028193A discloses a method and system for dynamic high-efficiency scheduling of big data tasks in a hybrid cloud platform. The method monitors the resources of tasks and worker nodes within the big data task hybrid cloud platform to obtain real-time resource utilization. Combined with task execution status, it employs a multi-dimensional scheduling strategy to schedule tasks, thereby achieving energy savings. Thus, the above method can adaptively adjust task allocation strategies, achieving efficient execution of offline tasks while reducing data center energy consumption. It can also dynamically optimize task allocation to maximize energy savings without affecting task execution. However, the above solution cannot achieve precise dynamic resource scheduling of big data platform computing tasks, thus failing to guarantee the efficiency of dynamic resource scheduling for big data platform computing tasks. Summary of the Invention

[0004] To address this issue, the present invention provides a method for dynamic resource scheduling of computing tasks on a big data platform, thereby overcoming the problem that existing technologies cannot achieve accurate dynamic resource scheduling of computing tasks on big data platforms, resulting in low efficiency in dynamic resource scheduling of computing tasks on big data platforms.

[0005] To achieve the above objectives, the present invention provides a method for dynamic resource scheduling of computing tasks on a big data platform, comprising: Receive submitted computing tasks and task resource requirement statements; Collect status data from several servers within the resource pool; The task queue is sorted and matched according to the task resource requirements and the status data; Send an occupation instruction to the matched server and distribute the corresponding computing task to the server; Monitor the real-time execution status of the server and add servers to the resource pool; After completing the computation task, clean up the resources used for the task computation and return them to the resource pool; The time required for the computation task to be completed from submission to completion is calculated. The resource dynamic scheduling of big data platform computing tasks is determined based on the task completion time. The dynamic resource scheduling of big data platform computing tasks is determined based on core dependency resources to determine whether it complies with standards. Alternatively, the reason why the dynamic resource scheduling of big data platform computing tasks does not meet the standards can be determined based on the proportion of waiting time. Alternatively, based on the judgment result, a corresponding processing instruction can be generated. Alternatively, it can determine the dynamic scheduling of resources for computing tasks on the big data platform. Adjust resource guarantee thresholds based on the type of core dependent resource servers, adjust CPU quotas based on CPU utilization differences, adjust memory quotas based on memory utilization, adjust corresponding quotas based on the type of high resource utilization, adjust scheduling priority weights, or adjust parallelism parameters.

[0006] Furthermore, the process of determining the dynamic resource scheduling of big data platform computing tasks based on the task completion time includes: When the task completion time is less than or equal to the first preset task completion time, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard, and the corresponding quota is adjusted based on the type of low resource utilization in the resource utilization rate. When the task completion time is greater than the first preset task completion time and less than or equal to the second preset task completion time, the dynamic resource scheduling of the big data platform computing task is determined based on the core dependent resources to determine whether it meets the standard. When the task completion time exceeds the second preset task completion time, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard, and the corresponding quota is adjusted based on the type of high resource utilization rate in the resource utilization rate.

[0007] Furthermore, the process of determining whether the dynamic resource scheduling of big data platform computing tasks conforms to the standard based on the core dependent resources includes: When all the core dependent resources of the critical server meet the standards, it is determined that the dynamic resource scheduling of the big data platform computing tasks meets the standards, and the determination of the dynamic resource scheduling of the big data platform computing tasks is completed. When the core dependent resources of the critical server do not all meet the standards, the resource guarantee threshold is adjusted based on the type of the core dependent resource server.

[0008] Furthermore, the process of adjusting the resource guarantee threshold based on the type of the core dependent resource server includes: When the core-dependent resource server is a high-core-dependent resource server, it is determined that the resource guarantee threshold be increased to a first multiple of the initial resource guarantee threshold; When the core-dependent resource server is a low-core-dependent resource server, it is determined that the resource guarantee threshold should be reduced to a second multiple of the initial resource guarantee threshold.

[0009] Furthermore, the process of adjusting the corresponding quota based on the type of low resource utilization rate in the resource utilization rate includes: When the type of low resource utilization in the resource utilization is CPU utilization, the CPU quota is adjusted based on the CPU utilization difference. When the type of low resource utilization in the resource utilization is memory utilization, the memory quota is adjusted based on the memory utilization.

[0010] Furthermore, the process of reducing the CPU quota based on the CPU utilization difference includes: The CPU quota is reduced based on the difference in CPU utilization, and the reduction in CPU quota is proportional to the difference in CPU utilization.

[0011] Furthermore, the process of reducing the memory quota based on the memory utilization rate includes: The memory quota is reduced based on the memory utilization rate, and the reduction in memory quota is inversely proportional to the memory utilization rate.

[0012] Furthermore, the process of adjusting the corresponding quota based on the type of high resource utilization rate in the resource utilization rate includes: When the type of high resource utilization in the resource utilization is CPU utilization, the CPU quota is increased to the third multiple of the initial CPU quota; When the type of high resource utilization in the resource utilization rate is memory utilization, the memory quota is increased to a fourth multiple of the initial memory quota.

[0013] Furthermore, the process of determining the dynamic resource scheduling of big data platform computing tasks based on the task completion time after the corresponding quota adjustment is completed includes: When the task completion time is less than or equal to the first preset task completion time, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard, and the corresponding quota is adjusted based on the type of low resource utilization in the resource utilization rate. When the task completion time is greater than the first preset task completion time and less than or equal to the second preset task completion time, the dynamic resource scheduling of the big data platform computing task is determined based on the core dependent resources to determine whether it meets the standard. When the task completion time exceeds the second preset task completion time, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard, and the reason for the failure to meet the standard is determined based on the waiting time ratio.

[0014] Furthermore, the process of determining why the dynamic resource scheduling of big data platform computing tasks does not meet standards based on the proportion of waiting time includes: When the percentage of waiting time is greater than the second preset percentage of waiting time, the scheduling priority weight is adjusted. When the percentage of waiting time is less than or equal to the second preset percentage of waiting time but greater than the first preset percentage of waiting time, the scheduling priority weight and parallelism parameters are adjusted.

[0015] When the percentage of waiting time is less than or equal to the first preset percentage of waiting time, the parallelism parameter is adjusted. The increase in scheduling priority weight is directly proportional to the proportion of waiting time, while the increase in parallelism parameter is inversely proportional to the proportion of waiting time.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention determines the dynamic resource scheduling of big data platform computing tasks based on task completion time, determines whether the dynamic resource scheduling of big data platform computing tasks meets the standard based on core dependent resources, determines the reason why the dynamic resource scheduling of big data platform computing tasks does not meet the standard based on the waiting time ratio, and generates corresponding processing instructions based on the determination results. It can timely and accurately determine whether the dynamic resource scheduling of big data platform computing tasks meets the standard and adjust the corresponding parameters. While effectively realizing the accurate dynamic resource scheduling of big data platform computing tasks, it also effectively improves the efficiency of dynamic resource scheduling of big data platform computing tasks.

[0017] Furthermore, this invention determines the dynamic resource scheduling of big data platform computing tasks based on task completion time, promptly and accurately identifying whether the corresponding quota needs to be adjusted based on the type of low resource utilization, whether the dynamic resource scheduling of big data platform computing tasks meets the standard based on core dependent resources, or whether the corresponding quota needs to be adjusted based on the type of high resource utilization. This avoids misjudgment and ensures judgment accuracy. While further realizing the accurate dynamic resource scheduling of big data platform computing tasks, it also further improves the efficiency of dynamic resource scheduling of big data platform computing tasks.

[0018] Furthermore, this invention determines whether the dynamic resource scheduling of big data platform computing tasks conforms to standards based on core dependent resources, accurately determines whether the resource guarantee threshold needs to be adjusted based on the type of core dependent resource server, further ensuring the accuracy of the determination. While further realizing the precise dynamic resource scheduling of big data platform computing tasks, it also further improves the efficiency of dynamic resource scheduling of big data platform computing tasks.

[0019] Furthermore, this invention adjusts the resource guarantee threshold based on the type of the core dependent resource server, effectively avoiding the situation where the precise dynamic scheduling of resources for big data platform computing tasks does not meet the standard due to the resource guarantee threshold not meeting the standard. While further realizing the precise dynamic scheduling of resources for big data platform computing tasks, it also further improves the efficiency of dynamic resource scheduling for big data platform computing tasks.

[0020] Furthermore, this invention adjusts the corresponding quota based on the type of low or medium resource utilization, and promptly determines whether the CPU quota needs to be adjusted based on the difference in CPU utilization or the memory quota needs to be adjusted based on memory utilization, thus avoiding misjudgment. While further realizing the precise dynamic scheduling of resources for computing tasks on the big data platform, it also further improves the efficiency of dynamic resource scheduling for computing tasks on the big data platform.

[0021] Furthermore, this invention reduces CPU quotas based on CPU utilization differences, effectively avoiding situations where the precise dynamic scheduling of resources for big data platform computing tasks fails to meet standards due to non-compliance with CPU quotas. This not only further achieves precise dynamic scheduling of resources for big data platform computing tasks but also further improves the efficiency of dynamic resource scheduling for big data platform computing tasks.

[0022] Furthermore, this invention reduces memory quotas based on memory utilization, effectively avoiding situations where the dynamic scheduling of resources for big data platform computing tasks fails to meet standards due to non-compliance with memory quotas. This not only further achieves precise dynamic scheduling of resources for big data platform computing tasks but also improves the efficiency of dynamic resource scheduling for big data platform computing tasks.

[0023] Furthermore, this invention adjusts the corresponding quotas based on the type of high resource utilization, increasing CPU quotas and memory quotas. This effectively avoids situations where the precise dynamic scheduling of resources for big data platform computing tasks fails to meet standards due to non-compliance of CPU and memory quotas. While further realizing precise dynamic scheduling of resources for big data platform computing tasks, it also further improves the efficiency of dynamic resource scheduling for big data platform computing tasks.

[0024] Furthermore, after the corresponding quota adjustment is completed, the present invention dynamically schedules resources for big data platform computing tasks based on task completion time, judges the adjustment effect of CPU quota and memory quota, and further determines the reasons why the dynamic scheduling of resources for big data platform computing tasks does not meet the standards. It promptly determines whether it is necessary to judge the dynamic scheduling of resources for big data platform computing tasks based on task completion time. While further realizing the accurate dynamic scheduling of resources for big data platform computing tasks, it also further improves the efficiency of dynamic scheduling of resources for big data platform computing tasks.

[0025] Furthermore, this invention determines the reasons why the dynamic resource scheduling of big data platform computing tasks does not meet the standards based on the proportion of waiting time, and determines that the scheduling priority weight or the parallelism parameter needs to be adjusted, or both the scheduling priority weight and the parallelism parameter need to be adjusted. While further realizing the accurate dynamic resource scheduling of big data platform computing tasks, it also further improves the efficiency of dynamic resource scheduling of big data platform computing tasks. Attached Figure Description

[0026] Figure 1 This is a system structure diagram of the resource dynamic scheduling method for computing tasks using a big data platform, as described in an embodiment of the present invention. Figure 2 This is a flowchart of a resource dynamic scheduling method for big data platform computing tasks according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating how to determine whether the dynamic resource scheduling of computing tasks on a big data platform conforms to standards, and the reasons for determining whether the standards are not met, according to an embodiment of the present invention. Figure 4 This is a flowchart illustrating the reasons why the dynamic resource scheduling of computing tasks on a big data platform does not meet the standards, as described in this embodiment of the invention. Detailed Implementation

[0027] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0028] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0029] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0030] Please see Figure 1 The diagram shown is a structural block diagram of a system for a resource dynamic scheduling method for computing tasks using a big data platform, according to an embodiment of the present invention. The system of this embodiment includes a receiving module, a monitoring module, a scheduling module, several servers, a management module, and an adjustment module; wherein, The receiving module is used to receive the submitted computing tasks and task resource requirement declarations; The monitoring module is connected to the receiving module and is used to collect status data of several servers in the resource pool. The scheduling module is connected to the receiving module and the monitoring module respectively, and is used to sort and match the task queue according to the task resource requirements and the status data; The scheduling module is also used to send an occupation instruction to the matching server and distribute the corresponding computing task to the server; The scheduling module is also used to monitor the real-time execution status of the server and add servers to the resource pool. The server is connected to the scheduling module and is used to clean up the resources used for task computation and return them to the resource pool after the computation task is completed. The management module is connected to the receiving module, the monitoring module, the scheduling module and the server respectively, and calculates the task completion time required from submission to completion of the computing task; The management module is also used to determine the dynamic resource scheduling of big data platform computing tasks based on the task completion time; The management module is also used to determine whether the dynamic resource scheduling of big data platform computing tasks conforms to the standards based on core dependent resources. Alternatively, the reason why the dynamic resource scheduling of big data platform computing tasks does not meet the standards can be determined based on the proportion of waiting time. Alternatively, based on the judgment result, a corresponding processing instruction can be generated. Alternatively, it can determine the dynamic scheduling of resources for computing tasks on the big data platform. The adjustment module is connected to the receiving module, the monitoring module, the scheduling module, the server, and the management module, respectively, and is used to adjust the resource guarantee threshold based on the type of the core dependent resource server, adjust the CPU quota based on the CPU utilization difference, adjust the memory quota based on the memory utilization, adjust the corresponding quota based on the type of high resource utilization, adjust the scheduling priority weight, or adjust the parallelism parameter.

[0031] Specifically, the task resource requirement declaration includes storage requirements, computing power requirements, task priority, and task type response level; The status data includes available server storage space, remaining CPU cores, free memory capacity, and network bandwidth utilization. The process of determining the type of the core dependency resource server based on the core dependency resource includes: Resource utilization includes storage utilization, CPU utilization, memory utilization, and network bandwidth utilization, calculated using the following formula: Storage utilization rate = (Total storage capacity - Available storage space) ÷ Total storage capacity × 100%; CPU utilization = (Total CPU cores - Remaining CPU cores) ÷ Total CPU cores × 100%; Memory utilization rate = (Total memory capacity - Free memory capacity) ÷ Total memory capacity × 100%; Network bandwidth utilization = Actual bandwidth used ÷ Total bandwidth × 100%; The resource utilization rate meets the standard when the storage utilization rate, the CPU utilization rate, the memory utilization rate, and the network bandwidth utilization rate all meet the standard. Wherein, the storage utilization rate meets the standard when it is less than or equal to the preset storage utilization rate A; the CPU utilization rate meets the standard when it is greater than or equal to the first preset CPU utilization rate B and less than or equal to the second preset CPU utilization rate D; the memory utilization rate meets the standard when it is greater than or equal to the third preset memory utilization rate E and less than or equal to the fourth preset memory utilization rate F; and the network bandwidth utilization rate meets the standard when it is less than or equal to the preset network bandwidth utilization rate H. In this embodiment, the preset storage utilization rate A = 90%, the first preset CPU utilization rate B = 30%, the second preset CPU utilization rate D = 90%, the third preset memory utilization rate E = 20%, the fourth preset memory utilization rate F = 90%, and the preset network bandwidth utilization rate H = 90%.

[0032] For a single computing task, the total number of servers used by the computing task is counted, and the servers used are divided into critical servers and ordinary servers according to the types of services they carry, and the total number of critical servers and ordinary servers is counted respectively. The core dependent resources of the critical server are determined according to the type of the critical server. When the critical server has both task scheduling and resource management functions, its core dependent resources are the CPU utilization and the memory utilization. When the critical server has both real-time data aggregation and calculation functions, its core dependent resources are the CPU utilization and the network bandwidth utilization. When the critical server has both metadata management and database functions, its core dependent resources are the storage utilization and the memory utilization. The critical servers identified in the system whose core dependency resources do not meet the standards are further categorized into high-core-dependency servers and low-core-dependency servers. A critical server is classified as a high-core-dependency server if its core dependency resources exceed a preset standard, and as a low-core-dependency server if they fall below the preset standard. Furthermore, when determining the type of a critical server, assuming it has both task scheduling and resource management functions, its core dependency resources are CPU utilization and memory utilization. If the determination based on CPU utilization differs from the determination based on memory utilization, the decision with higher priority is used. That is, if a server is classified as a high-core-dependency server, it is classified as such; otherwise, it is classified as a low-core-dependency server.

[0033] Please see Figure 2 The diagram shown is a flowchart of a resource dynamic scheduling method for computing tasks on a big data platform according to an embodiment of the present invention. The method described in this embodiment includes: Step S1: Receive the submitted computing task and task resource requirement declaration; Collect status data from several servers within the resource pool; Step S2: Sort and match the task queue according to the task resource requirements and the status data; Step S3: Send an occupation instruction to the matched server and distribute the corresponding computing task to the server; Monitor the real-time execution status of the server and add servers to the resource pool; Step S4: After completing the computing task, clean up the computing resources used for the task and return them to the resource pool; Step S5: Calculate the time required for the computation task to complete from submission. The resource dynamic scheduling of big data platform computing tasks is determined based on the task completion time. Step S6: Determine whether the dynamic resource scheduling of big data platform computing tasks conforms to the standard based on core dependent resources. Alternatively, the reason why the dynamic resource scheduling of big data platform computing tasks does not meet the standards can be determined based on the proportion of waiting time. Step S7, or, generate the corresponding processing instruction based on the determination result. Alternatively, it can determine the dynamic scheduling of resources for computing tasks on the big data platform. Steps S8-S9 involve adjusting the resource guarantee threshold based on the type of core dependent resource server, adjusting the CPU quota based on the CPU utilization difference, adjusting the memory quota based on the memory utilization, adjusting the corresponding quota based on the type of high resource utilization, adjusting the scheduling priority weight, or adjusting the parallelism parameter.

[0034] Please see Figure 3 The diagram shows a flowchart illustrating how an embodiment of the present invention determines whether the dynamic resource scheduling of a big data platform computing task conforms to a standard and the reasons for determining whether it does not conform to the standard. The process of determining the dynamic resource scheduling of a big data platform computing task based on the task completion time in this embodiment includes: When the task completion time is less than or equal to the first preset task completion time T1, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard, and the corresponding quota is adjusted based on the type of low resource utilization in the resource utilization rate. When the task completion time is greater than the first preset task completion time T1 and less than or equal to the second preset task completion time T2, the dynamic resource scheduling of the big data platform computing task is determined based on the core dependent resources to determine whether it meets the standard. When the task completion time exceeds the second preset task completion time T2, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard, and the corresponding quota is adjusted based on the type of high resource utilization in the resource utilization rate. Specifically, in this embodiment, if the CPU utilization and memory utilization are lower than a preset standard, they are recorded as low resource utilization, and the corresponding quotas are CPU quota and memory quota. If the CPU utilization and memory utilization are higher than a preset standard, they are recorded as high resource utilization.

[0035] Please continue reading. Figure 3 As shown, the process by which this embodiment of the invention determines whether the dynamic resource scheduling of big data platform computing tasks conforms to the standard based on the core dependent resources includes: When all the core dependent resources of the critical server meet the standards, it is determined that the dynamic resource scheduling of the big data platform computing tasks meets the standards, and the determination of the dynamic resource scheduling of the big data platform computing tasks is completed. When the core dependent resources of the critical server do not all meet the standards, the resource guarantee threshold is adjusted based on the type of the core dependent resource server. Specifically, adjusting the resource security threshold can prevent resource waste or instability.

[0036] Please continue reading. Figure 3 As shown, the process of adjusting the resource guarantee threshold based on the type of the core dependent resource server in this embodiment of the invention includes: When the core-dependent resource server is a high-core-dependent resource server, it is determined that the resource guarantee threshold should be increased to a first multiple of the initial resource guarantee threshold, wherein in this embodiment, the first multiple = 1.5 times; When the core-dependent resource server is a low-core-dependent resource server, it is determined that the resource guarantee threshold should be reduced to a second multiple of the initial resource guarantee threshold, wherein, in this embodiment, the second multiple = 0.5 times; Specifically, setting the values ​​to 1.5x and 0.5x ensures a balance between covering load requirements and avoiding resource waste.

[0037] Please continue reading. Figure 3 As shown, the process of adjusting the corresponding quota based on the type of low resource utilization rate in the resource utilization rate embodiment of the present invention includes: When the type of low resource utilization in the resource utilization is CPU utilization, the CPU quota is adjusted based on the CPU utilization difference. When the type of low resource utilization in the resource utilization is memory utilization, the memory quota is adjusted based on the memory utilization. Specifically, when both CPU utilization and memory utilization are of the low resource utilization type, both CPU quota and memory quota are adjusted; The CPU utilization difference is the difference between the first preset CPU utilization and the CPU utilization. As can be seen from the above, the first preset CPU utilization rate is 30%, and the CPU quota can be adjusted based on the difference in CPU utilization rate after actual debugging. Similarly, the third preset memory utilization rate is 20%, and the memory quota can be precisely adjusted directly based on the memory utilization rate after actual debugging.

[0038] Please see Figure 4The diagram shows a flowchart illustrating the reasons why the dynamic resource scheduling of computing tasks on a big data platform does not meet the standards, as described in this embodiment of the invention. The process of reducing the CPU quota based on the CPU utilization difference in this embodiment includes: When the CPU utilization difference is greater than or equal to the second preset CPU utilization difference ΔZ2, the CPU quota is reduced to 0.3 times the initial CPU quota. In this embodiment, the second preset CPU utilization difference ΔZ2 = 21.7%. When the CPU utilization difference is less than the second preset CPU utilization difference △Z2 and greater than or equal to the first preset CPU utilization difference △Z1, the CPU quota is reduced to 0.5 times the initial CPU quota. In this embodiment, the first preset CPU utilization difference △Z1 = 12.6%. When the CPU utilization difference is less than the first preset CPU utilization difference ΔZ1, the CPU quota is reduced to 0.7 times the initial CPU quota; Specifically, the larger the CPU utilization difference, the greater the reduction factor of the CPU quota. As mentioned above, the first preset CPU utilization is 30%, so the range of the CPU utilization difference is 0% to 30%. The reduction of the CPU quota is to release idle CPU computing power and allocate it to the computing tasks that need computing power. The values ​​of the CPU utilization difference and the reduction factor of the CPU quota are both derived from the actual debugging results.

[0039] Please continue reading. Figure 4 As shown, the process of reducing the memory quota based on the memory utilization rate in this embodiment of the invention includes: When the memory utilization rate is greater than or equal to the second preset memory utilization rate K2, the memory quota is reduced to 0.6 times the initial memory quota. In this embodiment, the second preset memory utilization rate K2 = 12.2%. When the memory utilization rate is less than the second preset memory utilization rate K2 and greater than or equal to the first preset memory utilization rate K1, the memory quota is reduced to 0.4 times the initial memory quota. In this embodiment, the first preset memory utilization rate K1 = 7.6%. When the memory utilization rate is less than the first preset memory utilization rate K1, the memory quota is reduced to 0.2 times the initial memory quota; Specifically, the lower the memory utilization rate, the greater the reduction factor of the memory quota. As mentioned above, the third preset memory utilization rate is 20%, so the value range of the memory utilization rate is 0% to 20%. The reduction of the memory quota is to release memory space and supply it to the computing tasks that need to be cached. The values ​​of the memory utilization rate and the reduction factor of the memory quota are both derived from the actual debugging results.

[0040] Please continue reading. Figure 4 As shown, the process of adjusting the corresponding quota based on the type of high resource utilization in the resource utilization rate of this embodiment of the invention includes: When the type of high resource utilization in the resource utilization is CPU utilization, the CPU quota is increased to the third multiple of the initial CPU quota, wherein in this embodiment, the third multiple = 1.5 times; When the type of high resource utilization in the resource utilization rate is memory utilization, the memory quota is increased to a fourth multiple of the initial memory quota, wherein in this embodiment, the fourth multiple = 2.0 times; Specifically, when both CPU utilization and memory utilization are of the high resource utilization type, both CPU quota and memory quota are adjusted; As can be seen from the above, the second preset CPU utilization rate is 90%, the fourth preset memory utilization rate is 90%, and the values ​​of 1.5 times and 2.0 times can ensure that after adjusting the corresponding quotas, the CPU utilization rate and the memory utilization rate remain within the standard range.

[0041] Please continue reading. Figure 4 As shown, the process of determining the dynamic resource scheduling of big data platform computing tasks based on the task completion time after the corresponding quota adjustment in this embodiment of the invention includes: When the task completion time is less than or equal to the first preset task completion time T1, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard, and the corresponding quota is adjusted based on the type of low resource utilization in the resource utilization rate. When the task completion time is greater than the first preset task completion time T1 and less than or equal to the second preset task completion time T2, the dynamic resource scheduling of the big data platform computing task is determined based on the core dependent resources to determine whether it meets the standard. When the task completion time exceeds the second preset task completion time T2, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard, and the reason for the failure to meet the standard is determined based on the waiting time ratio.

[0042] Specifically, the waiting time percentage is the ratio of the time spent waiting between submitting the computing task and starting to process it to the time taken to complete the task.

[0043] Please continue reading. Figure 4 As shown, the process of determining the reasons why the dynamic resource scheduling of big data platform computing tasks does not meet the standard based on the waiting time ratio in this embodiment of the invention includes: When the percentage of waiting time is greater than the second preset percentage of waiting time x2, the scheduling priority weight is adjusted and increased to 2.0 times the initial scheduling priority weight. In this embodiment, the second preset percentage of waiting time x2 = 50%. When the percentage of waiting time is less than or equal to the second preset waiting time percentage X2 and greater than the first preset waiting time percentage X1, the scheduling priority weight and parallelism parameter are adjusted. The scheduling priority weight is increased to 1.5 times the initial scheduling priority weight, and the parallelism parameter is increased to 1.5 times the initial parallelism parameter. In this embodiment, the percentage of waiting time is greater than the first preset waiting time percentage X1 = 30%.

[0044] When the percentage of waiting time is less than or equal to the first preset percentage of waiting time X1, the parallelism parameter is adjusted and increased to 2.0 times the initial parallelism parameter. Specifically, the percentage of waiting time, the multiple by which the scheduling priority weight is increased, and the multiple by which the parallelism parameter is increased are all derived from actual debugging results.

[0045] Example 1 For e-commerce platform user consumption behavior, the task queue includes real-time user behavior tracking tasks, real-time recommendation calculation tasks, real-time risk control identification tasks, real-time activity effect statistics tasks, offline user profile update tasks, consumption behavior path analysis tasks, product association analysis tasks, marketing activity effect review tasks, and abnormal behavior offline investigation tasks. For real-time recommendation tasks, the corresponding first preset task completion time is 100 ms, the second preset task completion time is 30 s, and the actual recommendation task completion time is 22 s, which is greater than the first preset task completion time of 100 ms and less than or equal to the second preset task completion time of 30 s. Based on the core dependent resources, it is determined whether the dynamic scheduling of resources for big data platform computing tasks meets the standards. When the core dependent resources of critical servers do not all meet the standards, the resource guarantee threshold is adjusted based on the type of core dependent resource server. If the core dependent resource server is a high core dependent resource server, the resource guarantee threshold is increased to 1.5 times the initial resource guarantee threshold.

[0046] Example 2 For operator-monitored network traffic, the task queue includes real-time traffic metric statistics tasks, real-time traffic anomaly detection tasks, real-time user experience monitoring tasks, real-time traffic scheduling triggering tasks, traffic trend analysis tasks, abnormal traffic tracing and review tasks, user behavior and traffic characteristic tasks, and network quality assessment tasks. For the real-time traffic anomaly detection task, the corresponding first preset task completion time is 110ms, the second preset task completion time is 25s, and the actual recommended task completion time is 92ms. If the task completion time is less than or equal to the first preset task completion time of 100ms, it is determined that the dynamic resource scheduling of the big data platform computing task does not meet the standard. The corresponding quota is adjusted based on the type of low resource utilization rate. The type of low resource utilization rate is CPU utilization. The CPU quota is adjusted based on the difference in CPU utilization rate. The first preset CPU utilization rate is 30%, the CPU utilization rate is 14%, and the CPU utilization rate difference is 16%. The CPU utilization rate difference is less than the second preset CPU utilization rate difference of 21.7% and greater than or equal to the first preset CPU utilization rate difference of 12.6%. The CPU quota is reduced to 0.5 times the initial CPU quota.

[0047] Example 3 For the integration of public service data into the government system, the task queue includes tasks for real-time response to public services, real-time monitoring of public service dynamics, real-time early warning of public service anomalies, data integration and archiving, analysis of the effectiveness of public service policies, prediction of public service demand trends, and compliance auditing of public service data. For the real-time response to public services task, the first preset task completion time is 300ms, and the second preset task completion time is 5 minutes. The actual recommended task completion time is 7.2 minutes, which exceeds the second preset task completion time by 5 minutes. Therefore, the resource dynamic scheduling of the big data platform's computing tasks is deemed non-compliant, and the corresponding quota is adjusted based on the type of high resource utilization. The highest resource utilization rate is CPU utilization. The CPU quota is increased to 1.5 times the initial CPU quota. After the corresponding quota adjustment, the resource dynamic scheduling of big data platform computing tasks is judged based on the task completion time. If the task completion time is 6 minutes, which is still greater than the second preset task completion time of 5 minutes, the resource dynamic scheduling of big data platform computing tasks is judged to be non-compliant. The reason for non-compliance of big data platform computing tasks is determined based on the waiting time ratio. If the waiting time ratio is 55%, which is greater than the second preset waiting time ratio of 50%, the scheduling priority weight is increased to 2.0 times the initial scheduling priority weight.

[0048] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0049] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for resource dynamic scheduling of computing tasks in a big data platform, characterized in that, The method comprises: receiving a submitted computing task and a task resource requirement declaration; collecting state data of a plurality of servers in a resource pool; sorting and matching a task queue according to the task resource requirement and the state data; sending an occupation instruction to a matched server and distributing the corresponding computing task to the server; monitoring the real-time execution state of the server and adding servers in the resource pool; cleaning up the resources used for the computing task and returning them to the resource pool after the computing task is completed; counting the task completion time required from submission to completion of the computing task; determining whether the resource dynamic scheduling of the computing task of the big data platform meets the standard based on the task completion time; determining whether the resource dynamic scheduling of the computing task of the big data platform meets the standard based on the core dependent resource, or determining the reason why the resource dynamic scheduling of the computing task of the big data platform does not meet the standard based on the waiting time proportion, or generating a corresponding processing instruction based on the determination result, or completing the determination of the resource dynamic scheduling of the computing task of the big data platform; adjusting the resource guarantee threshold based on the type of the core dependent resource server, adjusting the CPU quota based on the CPU utilization difference, adjusting the memory quota based on the memory utilization, adjusting the corresponding quota based on the type of the medium and high resource utilization in the resource utilization, adjusting the scheduling priority weight, or adjusting the parallelism parameter. 2.The method of claim 1, wherein, The process of determining the resource dynamic scheduling of the computing task of the big data platform based on the task completion time comprises: when the task completion time is less than or equal to a first preset task completion time, determining that the resource dynamic scheduling of the computing task of the big data platform does not meet the standard, and adjusting the corresponding quota based on the type of the low resource utilization in the resource utilization; when the task completion time is greater than the first preset task completion time and less than or equal to a second preset task completion time, determining whether the resource dynamic scheduling of the computing task of the big data platform meets the standard based on the core dependent resource; when the task completion time is greater than the second preset task completion time, determining that the resource dynamic scheduling of the computing task of the big data platform does not meet the standard, and adjusting the corresponding quota based on the type of the medium and high resource utilization in the resource utilization. 3.The method of claim 2, wherein, The process of determining whether the resource dynamic scheduling of the computing task of the big data platform meets the standard based on the core dependent resource comprises: when the core dependent resources of the key servers all meet the standard, determining that the resource dynamic scheduling of the computing task of the big data platform meets the standard, and completing the determination of the resource dynamic scheduling of the computing task of the big data platform; when the core dependent resources of the key servers do not all meet the standard, adjusting the resource guarantee threshold based on the type of the core dependent resource server.

4. The method of claim 3, wherein, The process of adjusting the resource guarantee threshold based on the type of the core dependent resource server comprises: when the core dependent resource server is a high core dependent resource server, determining to increase the resource guarantee threshold to a first multiple of an initial resource guarantee threshold; when the core dependent resource server is a low core dependent resource server, determining to decrease the resource guarantee threshold to a second multiple of the initial resource guarantee threshold.

5. The method of claim 2, wherein, The process of adjusting the corresponding quota based on the type of the low resource utilization in the resource utilization includes: When the type of the low resource utilization in the resource utilization is the CPU utilization, adjusting the CPU quota based on the CPU utilization difference; When the type of the low resource utilization in the resource utilization is the memory utilization, adjusting the memory quota based on the memory utilization.

6. The method of claim 5, wherein, The process of reducing the CPU quota based on the CPU utilization difference includes: The CPU quota is reduced based on the CPU utilization difference, and the reduction amplitude of the CPU quota is proportional to the CPU utilization difference.

7. The method of claim 6, wherein, The process of reducing the memory quota based on the memory utilization includes: The memory quota is reduced based on the memory utilization, and the reduction amplitude of the memory quota is inversely proportional to the memory utilization. 8.The method of claim 2, wherein, The process of adjusting the corresponding quota based on the type of the high resource utilization in the resource utilization includes: When the type of the high resource utilization in the resource utilization is the CPU utilization, increasing the CPU quota to a third multiple of the initial CPU quota; When the type of the high resource utilization in the resource utilization is the memory utilization, increasing the memory quota to a fourth multiple of the initial memory quota. 9.The method of claim 8, wherein, The process of determining the resource dynamic scheduling of the big data platform computing task based on the task completion time after the corresponding quota is adjusted includes: When the task completion time is less than or equal to a first preset task completion time, determining that the resource dynamic scheduling of the big data platform computing task does not meet the standard, and adjusting the corresponding quota based on the type of the low resource utilization in the resource utilization; When the task completion time is greater than the first preset task completion time and less than or equal to a second preset task completion time, determining whether the resource dynamic scheduling of the big data platform computing task meets the standard based on the core dependent resource; When the task completion time is greater than the second preset task completion time, determining that the resource dynamic scheduling of the big data platform computing task does not meet the standard, and determining the reason why the resource dynamic scheduling of the big data platform computing task does not meet the standard based on the waiting time proportion. 10.The method of claim 9, wherein, The process of determining the reason why the resource dynamic scheduling of the big data platform computing task does not meet the standard based on the waiting time proportion includes: When the waiting time proportion is greater than a second preset waiting time proportion, adjusting the scheduling priority weight; When the waiting time proportion is less than or equal to the second preset waiting time proportion and greater than a first preset waiting time proportion, adjusting the scheduling priority weight and the parallelism parameter; When the waiting time proportion is less than or equal to the first preset waiting time proportion, adjusting the parallelism parameter; The increase amplitude of the scheduling priority weight is proportional to the waiting time proportion, and the increase amplitude of the parallelism parameter is inversely proportional to the waiting time proportion.

Images