Spark SQL task scheduling optimization method based on dynamic resources

Abstract

The invention discloses a Spark SQL (Structured Query Language) task scheduling optimization method based on dynamic resources, which is characterized in that a job submitted by a user is divided into a plurality of stages capable of being executed in parallel, and each task set comprises a plurality of tasks; presetting an attenuation median, and dividing each stage into a large stage and a small stage; constructing an attenuation function based on the attenuation median, and calculating an attenuation factor; for each stage to be subjected to resource allocation, based on an attenuation function, an attenuation factor and the total number of residual resources, performing preliminary resource allocation for each stage to be subjected to resource allocation; and performing normalization and upper limit protection setting on the initial resource allocation of each stage of the resources to be allocated, obtaining real resources allocated to each stage of the resources to be allocated, and completing task execution scheduling and resource allocation. According to the method, the problem that other tasks wait for a long time due to the fact that part of large tasks seize resources under the high-load condition due to the fact that task scheduling in an existing Spark cluster is not limited is solved.

Description

Technical Field

[0001] This invention relates to the fields of big data analysis and real-time computing, specifically to a Spark SQL task scheduling optimization method based on dynamic resources. Background Technology

[0002] In the field of big data processing and distributed computing, Apache Spark, as a mainstream general-purpose distributed computing framework, is widely used in scenarios such as batch processing, streaming computing, and interactive analysis of massive amounts of data. Its core task scheduling mechanism is based on a layered architecture design, specifically including: the DAGScheduler divides user-submitted jobs into multiple stages that can be executed in parallel, each stage corresponding to a TaskSet, and then submits the TaskSet to the TaskScheduler for task distribution and execution scheduling. This scheduling hierarchy can be summarized as: Job → Stage → TaskSet → Task.

[0003] In typical application scenarios involving concurrent execution of multiple tasks, Spark provides two mainstream task scheduling strategies: First-In-First-Out (FIFO) and Fair Scheduling (FAIR). FIFO schedules tasks sequentially according to their submission order and is suitable for scenarios where task priorities are not significantly different. FAIR scheduling, on the other hand, uses a resource quota allocation mechanism to attempt to achieve a balanced distribution of resources among multiple concurrent jobs, thereby improving overall system throughput and response fairness.

[0004] However, existing scheduling mechanisms still have significant limitations when facing dynamic load fluctuations in resource-constrained environments. Especially in Spark SQL services, when the number of Executor nodes in the Spark cluster has reached the system configuration limit, i.e., resources are saturated, existing scheduling strategies struggle to effectively address the following key issues:

[0005] The problem of resource "monopoly" for large tasks: Some complex SQL queries (such as those involving non-equi-joins between large tables, full table scans, and large-scale aggregations) generate a large number of TaskSets, which contain hundreds or thousands of Tasks. In FAIR scheduling mode, although each Job is assigned an initial resource quota, once such large tasks start executing, they may occupy a large amount of Executor resources for a long time, causing their respective TaskSets to be unable to release resources for an extended period, thereby blocking the scheduling of TaskSets for other concurrent Jobs.

[0006] The phenomenon of small tasks "starving" is frequent: Because TaskScheduler does not dynamically perceive and differentiate the resource consumption characteristics of tasks within TaskSet, the system cannot identify and limit the preemptive behavior of "resource-greedy" tasks. When multiple small tasks or high-priority tasks are waiting for resources, large tasks cannot be released in time due to continuous resource occupation, resulting in an imbalance in resource utilization and seriously affecting the overall responsiveness and fairness of the system.

[0007] Static scheduling strategies lack adaptability: Current FIFO and FAIR scheduling strategies are based on static or semi-static resource allocation logic, which do not fully consider key factors such as dynamic resource demand changes during task execution, task execution progress, and resource release rhythm. Therefore, it is difficult to achieve refined and intelligent task scheduling under resource-scarce conditions.

[0008] The aforementioned problems are particularly prominent in real-world production environments, especially in high-concurrency Spark SQL service scenarios, where the phenomenon of "large tasks freezing the system and small tasks queuing for extended periods" frequently occurs, severely restricting the system's concurrent processing capabilities and user experience. Therefore, there is an urgent need for an optimization method that can dynamically perceive task resource consumption characteristics and intelligently adjust task scheduling priorities under resource-constrained conditions, in order to address the fairness and efficiency bottlenecks of existing scheduling mechanisms in resource contention scenarios. Summary of the Invention

[0009] The purpose of this invention is to provide a Spark SQL task scheduling optimization method based on dynamic resources. This method aims to solve the problem that the unrestricted task scheduling in existing Spark clusters may lead to some large tasks preempting resources under high load, causing other tasks to wait for a long time.

[0010] To achieve the above functions, this invention designs a Spark SQL task scheduling optimization method based on dynamic resources. Based on the Spark SQL framework, the following steps S1-S5 are executed to complete task execution scheduling and resource allocation: Step S1: When the Spark SQL framework's Driver service starts, a periodic task for detecting Spark cluster load information is started. The load of the Spark cluster is detected every preset time interval. If the current Spark cluster is in a high-load state, proceed to step S2. Step S2: For the user-submitted job, divide it into multiple stages that can be executed in parallel. Each stage corresponds to a task set, and each task set contains multiple tasks. Preset the median decay value, and divide each stage into a large stage and a small stage respectively. Proceed to step S3. Step S3: Based on the median attenuation, construct the attenuation function and calculate the attenuation factor; proceed to step S4; Step S4: For each stage of resources to be allocated, based on the decay function, decay factor, and the total number of remaining resources, perform preliminary resource allocation for each stage of resources to be allocated; proceed to Step S5. Step S5: For the initial resource allocation of each stage to be allocated resources, normalize and set upper limit protection to obtain the actual resources allocated to each stage to be allocated resources, and complete the task execution scheduling and resource allocation.

[0011] As a preferred technical solution of the present invention: In step S1, if the number of currently running tasks or the total number of task slots in the Spark cluster exceeds a preset ratio, the Spark cluster is determined to be in a state of instantaneous high load in the current detection period; if the Spark cluster is in a state of instantaneous high load for three consecutive detection periods, the Spark cluster is determined to be in a state of high load.

[0012] As a preferred technical solution of the present invention, the method for dividing the large stage and the small stage in step S2 is as follows: Assuming the current SQL instance can provide a total of N tasks, define the median decay M and the median decay coefficient C, where M = C * N. The median decay M represents the maximum number of tasks that the current SQL instance can tolerate in a phase. A phase with no more than the median decay M is a small phase, and a phase with more than the median decay M is a large phase.

[0013] As a preferred technical solution of the present invention: In step S3, the method for constructing the attenuation function is as follows: A small stage is defined as a stage with the number of tasks in the range (0, C*N], and a large stage is defined as a stage with the number of tasks in the range (0, C*N]. Within the range of stages, the decay function is constructed as follows: , in, This represents the decay factor for resource throttling at the target stage. This represents the boundary value of the decay factor between the small and large stages. This indicates the total number of tasks in the currently scheduled phase. This represents the median decay coefficient. This indicates that the total number of tasks that the current SQL instance can provide is N.

[0014] According to claim 4, a Spark SQL task scheduling optimization method based on dynamic resources is characterized in that, in the decay function constructed in step S3, when x = C*N, the decay factor is... ,Pick .

[0015] As a preferred embodiment of the present invention, the specific steps of step S4 are as follows: Step S4.1: Assume that at the current time T, the total number of remaining resources in the Spark cluster is R, and the number of stages awaiting resource allocation is m-1; the sequence of resource requirements for all stages awaiting resource allocation at time T is as follows: Meanwhile, the decay factor corresponding to each stage is: The allocatable resources for each stage of the i-th resource allocation process. As shown in the following formula: , In the formula, Let represent the decay factor for each stage of the i-th resource to be allocated. This represents the sequence of resource requirements for each stage of resource allocation. This represents the number of resources required in the i-th stage after decay by the decay factor, where ;k represents the k stages with the fewest tasks selected after sorting the tasks in each stage in ascending order. Step S4.2: Needle Metrics, setting thresholds Secondary intervention in resource allocation is carried out, as follows: when At this time, only the stage with the fewest number of tasks among the first k stages is selected for resource allocation. ; when When resources are allocated directly to the m stages waiting for allocation, the selection is made at this time. ; After secondary intervention in resource allocation, assuming the system involves k stages of resource allocation at time T, the sequence of resource requirements for the stages to be allocated after system selection at time T is as follows: .

[0016] As a preferred technical solution of the present invention: in step S4, when At that time, the task resources required for the m stages are sorted in ascending order, and k is the minimum value that satisfies the following constraints: , In the formula, R represents the sequence of resource requirements for each stage of the resource allocation process, and R represents the total number of remaining resources in the Spark cluster at the current time T.

[0017] As a preferred technical solution of the present invention: in step S4, the minimum number of task resources to be allocated is forced to be 1.

[0018] As a preferred technical solution of the present invention: In step S5, the method for normalizing and setting upper limit protection for the preliminary resource allocation of each stage to be allocated resources is as follows: The actual resources allocated in the first k stages As shown in the following formula: , In the formula, This represents the total number of remaining resources in the Spark cluster at the current time T; This indicates the upper limit of resource allocation set for each stage; This represents the allocatable resources for the i-th stage where resources are to be allocated.

[0019] Beneficial effects: Compared with the prior art, the advantages of the present invention include: 1. Implement dynamic resource regulation by "limiting traffic for large tasks and prioritizing small tasks": This invention introduces a decay factor mechanism based on the number of tasks to implement exponential resource decay for large stages, significantly suppressing their resource preemption behavior; at the same time, it maintains low decay or even no decay for small stages, realizing resource allocation to small tasks and effectively solving the problem of "large tasks monopolizing resources".

[0020] 2. Possesses high load adaptability to prevent resource fragmentation and scheduling blockage: By using periodic load detection and a "three-cycle" judgment mechanism, the rate limiting strategy is only activated when there is a real high load. Combined with resource fragmentation detection and truncation mechanism, stages with low resource requirements are prioritized for scheduling, avoiding a large number of small tasks from occupying space, thereby improving scheduling efficiency and system throughput.

[0021] 3. The resource allocation process is configurable and controllable, ensuring system stability and fairness:

[0022] A dynamic allocation upper limit constraint and a normalized allocation mechanism are introduced to prevent a single large task from monopolizing all resources; all parameters (such as the decay median coefficient, decay coefficient, and sampling period) are configurable to adapt to Spark clusters of different sizes and business scenarios, balancing flexibility and robustness. Attached Figure Description

[0023] Figure 1 This is a flowchart of a Spark SQL task scheduling optimization method based on dynamic resources according to an embodiment of the present invention; Figure 2 This is a graph showing the value of the attenuation function provided according to an embodiment of the present invention; Figure 3 This is a diagram showing the task slot allocation for a phase of task scheduling in a partial scenario of an actual running task, provided by an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0025] In this embodiment, the terminology used is explained as follows: Management node (Spark Driver): The management node for Spark SQL programs; Executor: Refers to the specific execution node of a task running in Spark; TaskScheduler: The service unit in Spark that schedules and executes task sets (TaskSets). It is responsible for scheduling the tasks (Tasks) in the TaskSet generated by DAGScheduler to the corresponding execution nodes according to the corresponding scheduling strategy. Stage: A logical stage in the execution of a Spark job, representing a set of tasks with dependencies. A TaskSet is a collection of tasks generated by the Stages that a Job is split into during runtime in Spark. Each Stage corresponds to one TaskSet, and each TaskSet contains one or more Tasks. It is an object used by the TaskScheduler for task scheduling.

[0026] The existing native Spark SQL FAIR scheduling mechanism has the following drawbacks: 1. Lack of a dynamic mechanism to suppress the resource consumption of large tasks:

[0027] While Spark's native FAIR scheduling implements resource quota allocation at the Job level, it does not perceive or hierarchically control the number of Tasks within a TaskSet. When a large task (such as a large table non-equi-join) generates a large number of Tasks, these Tasks can continue to preempt and occupy Executor resources for extended periods, causing other tasks to wait indefinitely due to insufficient resources, resulting in "resource monopoly" and "task starvation."

[0028] 2. The resource allocation strategy is static and lacks flexibility, making it difficult to cope with high-load scenarios: The native scheduling mechanism uses a fixed or semi-static resource allocation method, which cannot dynamically adjust the resource acquisition limit of each TaskSet according to the instantaneous load of the Spark cluster. When resources are saturated, the system cannot effectively suppress resource requests from large tasks, and it also lacks a mechanism to prevent resource fragmentation, resulting in decreased scheduling efficiency and reduced overall throughput.

[0029] 3. Intelligent matching of task size and priority is not implemented: The native scheduling treats all TaskSets equally, without differentiating them based on task size (number of tasks). Small tasks, due to their low resource requirements, should be prioritized for scheduling, but they are often "crowded out" by large tasks, failing to guarantee the response speed of short, high-priority tasks and impacting system real-time performance and user experience.

[0030] To address the aforementioned shortcomings, the native Spark SQL FAIR scheduling mechanism has been modified and optimized. Under high load scenarios in the Spark cluster, the maximum number of task slots each job can acquire is limited. This ensures that all tasks have a chance to obtain computing resources and prevents other tasks from waiting for extended periods due to resource contention by some large tasks. The core points of the improved rate limiting mechanism are as follows: 1. Add a load check mechanism:

[0031] When the Spark SQL Driver service starts, a periodic task for monitoring Spark cluster load information is also started, which checks the cluster load every 1 minute by default (configurable). If the ratio of "currently running tasks / total number of task slots in the cluster" exceeds a certain percentage, such as 90% (configurable), it is determined to be a current instantaneous high load state.

[0032] 2. When to implement traffic restrictions: If three consecutive status detection cycles show a momentary high load, the current Spark cluster is considered to be in a high load state, and only then can the task scheduling resource rate limiting mechanism be enabled.

[0033] 3. Traffic limiting method: Modify Spark's TaskScheduler implementation to ensure that the number of tasks obtained by each TaskSet to be scheduled does not exceed a dynamically calculated threshold during TaskSet scheduling. This limits the resource allocation of the current TaskSet and avoids excessive resource contention.

[0034] To find a suitable threshold, we analyzed the distribution of task numbers in each Stage in some real-world business scenarios and proposed the concept of a decay factor. The decay factor is defined as the proportion of task slots allocated to a Stage relative to the remaining allocable task slots. Taking into account factors such as the number of tasks in a Stage, the number of allocated task slots, the total number of task slots in the Spark cluster, the number of remaining task slots, and the number of Stages waiting to be scheduled, we established a decay curve showing the decay factor changing with the number of tasks in a Stage. Based on the decay factor, remaining resources, and the number of tasks in all Stages waiting to be scheduled, we calculated the number of slots that the TaskSet could be allocated for this scheduling.

[0035] By employing the above full-stage task scheduling resource throttling strategy, the following effects are expected to be achieved: (1) The fewer remaining task resources in the Spark cluster (the higher the load), the fewer task slots are allocated to each stage to be scheduled.

[0036] (2) When resources are insufficient, Stages with smaller resource requirements are scheduled to have a higher priority than Stages with larger resource requirements, so that the resource requirements of Stages with smaller requirements are met first.

[0037] A Spark SQL task scheduling optimization method based on dynamic resources, characterized by being based on the Spark SQL framework, referring to... Figure 1 Perform the following steps S1-S5 to complete task execution scheduling and resource allocation: Step S1: When the Spark SQL framework's Driver service starts, a periodic task for detecting Spark cluster load information is started. The load of the Spark cluster is detected every preset time interval. In this example, the preset time interval is 1 minute. If the current Spark cluster is in a high-load state, proceed to step S2. In step S1, if the number of currently running tasks or the total number of task slots in the Spark cluster exceeds a preset ratio (in this embodiment, the preset ratio is 90%), then the Spark cluster is determined to be in a state of instantaneous high load during the current detection period. If the Spark cluster is in a state of instantaneous high load for three consecutive detection periods, then the Spark cluster is determined to be in a state of high load.

[0038] Step S2: Divide the user-submitted job into multiple stages that can be executed in parallel. Each stage corresponds to a task set, and each task set contains multiple tasks. Preset the median decay value and divide each stage into large stages and small stages respectively. Proceed to step S3. To more rationally allocate limited task resources to appropriate stages and improve system throughput, stages are divided into two types: small stages and large stages. The current partitioning algorithm only considers the number of tasks required to complete each stage.

[0039] The method for dividing into major and minor stages is as follows: Assume the current SQL instance can provide a total of N tasks, where N is only related to the resources configured for that SQL instance. Define the median decay M and the median decay coefficient C, where M = C * N. The median decay M represents the maximum number of tasks the current SQL instance can tolerate in a given phase. Phases with no more than the median decay M are considered small phases, while phases with more than the median decay M are considered large phases. When allocating task resources, rewards or mild decay are applied. For large phases, task resource allocation will involve a more severe and significant decay than in small phases.

[0040] Step S3: Based on the median attenuation, construct the attenuation function and calculate the attenuation factor; proceed to step S4; The method for constructing the decay function is as follows: A small stage is defined as a stage with the number of tasks in the range (0, C*N], and a large stage is defined as a stage with the number of tasks in the range (0, C*N]. Within the range of stages, the decay function is constructed as follows: , in, This represents the decay factor for resource throttling at the target stage. This represents the boundary value of the decay factor between the small and large stages. This indicates the total number of tasks in the currently scheduled phase. This represents the median decay coefficient. This indicates that the total number of tasks that the current SQL instance can provide is N.

[0041] in This indicates the magnitude by which the number of tasks in a stage deviates from the median decay. This indicates the percentage by which the number of tasks in a phase deviates from the median decay. The decay factor for a smaller phase is not less than [a certain percentage]. The decay factor of the large stage is less than When x = C * N, the attenuation factor ,according to The practical significance of taking .

[0042] In one embodiment, taking a task actually running on a certain business line as an example, the number of tasks in each stage of all jobs running online within a certain week is counted. Assuming... If the median attenuation coefficient C = 5 (which can be matched), then its attenuation function is as follows: , In the above embodiments, the attenuation function curve is referenced. Figure 2 .

[0043] Step S4: For each stage of resources to be allocated, based on the decay function, decay factor, and the total number of remaining resources, perform preliminary resource allocation for each stage of resources to be allocated; proceed to Step S5. The specific steps of step S4 are as follows: Step S4.1: Assume that at the current time T, the total number of remaining resources in the Spark cluster (SQL analysis instance) is R, and the number of stages with resources to be allocated is m-1; the sequence of resource requirements for all stages with resources to be allocated at time T is as follows: Meanwhile, the decay factor corresponding to each stage is: To prevent the number of task resources obtained in a super-large stage from decaying to zero and thus becoming unavailable, the minimum number of task resources allocated is forced to be 1. The allocatable resources for each stage awaiting resource allocation are as follows: As shown in the following formula: , In the formula, Let represent the decay factor for each stage of the i-th resource to be allocated. This represents the sequence of resource requirements for each stage of resource allocation. This represents the number of resources required in the i-th stage after decay by the decay factor, where ;k represents the k stages with the fewest tasks selected after sorting the tasks in each stage in ascending order. This represents the expected number of task resources allocated at time T in the i-th stage. This ensures that the number of tasks does not exceed the average. It also significantly reduces the impact of the super-large phase. This prevents resources from being allocated during the super-large phase while ensuring that resources are not fragmented.

[0044] Step S4.2: This metric represents the scenario where the system's remaining resources at time T are evenly distributed among stages waiting for resource allocation. A smaller value indicates a more dispersed resource allocation, potentially causing most tasks to run slowly. To ensure resources are tilted towards smaller tasks, [further measures are needed]. Metrics, setting thresholds Secondary intervention in resource allocation is carried out, as follows: when This indicates that system resources may be fragmented. Therefore, it is necessary to truncate the allocation sequence, ensuring that only stages with fewer resources participate in the allocation. In this case, only the stage with the fewest k tasks is selected for resource allocation. ; when When the probability of fragmented allocation of system resources is low, resource allocation is directly performed on the m stages waiting for allocation. In this case, [the appropriate option is to choose...]. ; After secondary intervention in resource allocation, assuming the system's resource allocation tasks at time T consist of k stages (k is not necessarily equal to m), the sequence of resource demands for the stages awaiting allocation after system selection at time T is as follows: .

[0045] when At this time, system resources may be fragmented. Sort the required task resources for the m stages in ascending order, and select the smallest stage sequence that satisfies the condition that the total number of tasks to be allocated is not less than R. That is, k is the minimum value that satisfies the following constraints: , In the formula, R represents the sequence of resource requirements for each stage of resource allocation, and R represents the total number of remaining resources in the cluster at the current time T.

[0046] Step S5: For the initial resource allocation of each stage to be allocated resources, normalize and set upper limit protection to obtain the actual resources allocated to each stage to be allocated resources, and complete the task execution scheduling and resource allocation.

[0047] The above series of operations imposes strong constraints on the expected resource allocation for each stage. However, considering that at time T, the total number of remaining resources in the Spark cluster is R, the resources that each stage can ultimately allocate need to be normalized. The method for normalizing the initial resource allocation for each stage to be allocated resources and setting an upper limit protection is as follows:

[0048] The actual resources allocated in the first k stages As shown in the following formula: , In the formula, This represents the total number of remaining resources in the Spark cluster at the current time T; This indicates the upper limit of resource allocation set for each stage; This represents the allocatable resources for the i-th stage where resources are to be allocated.

[0049] Considering that when there is only one stage waiting to be scheduled, i.e., m=1, and the number of tasks required for that stage is very large, i.e. This would cause all resources in the Spark cluster to be allocated to an excessively large stage, resulting in subsequent scheduled stages waiting for a long time without available task resources. To solve this problem, an upper limit U for resource allocation is increased for each stage, with U=R / 2 by default.

[0050] Taking the task slot allocation of a certain scenario in the actual operation of a business line as an example, the test results are as follows: Figure 3 As shown, through Figure 3 The test results show that the task scheduling and rate limiting mechanism designed based on this invention basically achieves the goal of limiting the allocation of resources for large tasks and prioritizing the allocation of resources for small tasks when the system is under high load, effectively avoiding the situation where small tasks "starve".

[0051] In summary, the method designed in this invention is implemented by modifying Spark TaskScheduler and adding a dynamic rate limiting mechanism for task scheduling. Based on information such as currently allocated resources, remaining resources, and total resources, the number of task slots allocated in each stage of a single job is dynamically limited, thereby achieving the purpose of limiting the allocation of large task resources and prioritizing the allocation of small task resources when the system is under high load.

[0052] The core implementation is divided into three levels:

[0053] (1) Selection of the timing of rate limiting: By adding a system load detection mechanism, the system load is detected regularly. The above task scheduling rate limiting mechanism can be started when the system is under high load.

[0054] (2) Design of decay function: The size of the task is determined by the number of tasks in the stage. Through the design and deduction of decay algorithm, the value curve of decay function required for the number of scheduling tasks is determined.

[0055] (3) Optimization of scheduling algorithm: Following the derivation of the decay algorithm, modify the implementation of Spark TaskScheduler task scheduling service, add the decay algorithm to the calculation logic of the scheduling task limit, so as to limit the resources of large tasks and give priority to the allocation of small task resources under high load.

[0056] The method designed in this invention identifies the load status of the Spark cluster and enables a resource allocation rate limiting mechanism for large tasks when under high load, so as to limit the allocation of resources for large tasks and prioritize the allocation of resources for small tasks.

[0057] This invention derives and designs a decay function, and implements the decay algorithm in Spark task scheduling. The algorithm dynamically calculates the resource allocation constraints for the stage task scheduling of each job.

[0058] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A Spark SQL task scheduling optimization method based on dynamic resources, characterized in that, Based on the Spark SQL framework, execute steps S1-S5 to complete task execution scheduling and resource allocation: Step S1: When the Spark SQL framework's Driver service starts, a periodic task for detecting Spark cluster load information is started. The load of the Spark cluster is detected every preset time interval. If the current Spark cluster is in a high-load state, proceed to step S2. Step S2: For the user-submitted job, divide it into multiple stages that can be executed in parallel. Each stage corresponds to a task set, and each task set contains multiple tasks. The median decay value is preset, and each stage is divided into a large stage and a small stage for each stage. Proceed to step S3; Step S3: Based on the median attenuation, construct the attenuation function and calculate the attenuation factor; Proceed to step S4; Step S4: For each stage of resources to be allocated, based on the decay function, decay factor, and the total number of remaining resources, perform preliminary resource allocation for each stage of resources to be allocated. Proceed to step S5; Step S5: For the initial resource allocation of each stage to be allocated resources, normalize and set upper limit protection to obtain the actual resources allocated to each stage to be allocated resources, and complete the task execution scheduling and resource allocation.

2. The Spark SQL task scheduling optimization method based on dynamic resources according to claim 1, characterized in that, In step S1, if the number of currently running tasks or the total number of task slots in the Spark cluster exceeds a preset ratio, the Spark cluster is determined to be in a state of instantaneous high load in the current detection period; if the Spark cluster is in a state of instantaneous high load for three consecutive detection periods, the Spark cluster is determined to be in a state of high load.

3. The Spark SQL task scheduling optimization method based on dynamic resources according to claim 1, characterized in that, The method for dividing the large and small stages in step S2 is as follows: Assuming the current SQL instance can provide a total of N tasks, define the median decay M and the median decay coefficient C, where M = C * N. The median decay M represents the maximum number of tasks that the current SQL instance can tolerate in a phase. A phase with no more than the median decay M is a small phase, and a phase with more than the median decay M is a large phase.

4. The Spark SQL task scheduling optimization method based on dynamic resources according to claim 3, characterized in that, In step S3, the method for constructing the decay function is as follows: A small stage is defined as a stage with the number of tasks in the range (0, C*N], and a large stage is defined as a stage with the number of tasks in the range (0, C*N]. Within the range of stages, the decay function is constructed as follows: ， in, This represents the decay factor for resource throttling at the target stage. This represents the boundary value of the decay factor between the small and large stages. This indicates the total number of tasks in the currently scheduled phase. This represents the median decay coefficient. This indicates that the total number of tasks that the current SQL instance can provide is N.

5. The Spark SQL task scheduling optimization method based on dynamic resources according to claim 4, characterized in that, In the attenuation function constructed in step S3, when x = C*N, the attenuation factor is... ,Pick .

6. The Spark SQL task scheduling optimization method based on dynamic resources according to claim 1, characterized in that, The specific steps of step S4 are as follows: Step S4.1: Assume that at the current time T, the total number of remaining resources in the Spark cluster is R, and the number of stages with resources to be allocated is m; the sequence of resource requirements for all stages with resources to be allocated at time T is as follows: Meanwhile, the decay factor corresponding to each stage is: The allocatable resources for each stage of the i-th resource to be allocated As shown in the following formula: ， In the formula, Let represent the decay factor for each stage of the i-th resource to be allocated. This represents the sequence of resource requirements for each stage of resource allocation. This represents the number of resources required in the i-th stage after decay by the decay factor, where ;k represents the k stages with the fewest tasks selected after sorting the tasks in each stage in ascending order. Step S4.2: For Metrics, setting thresholds Secondary intervention in resource allocation is carried out, as follows: when At this time, only the stage with the fewest number of tasks among the first k stages is selected for resource allocation. ; when When resources are allocated directly to the m stages waiting for allocation, the selection is made at this time. ; After secondary intervention in resource allocation, assuming the system involves k stages of resource allocation at time T, the sequence of resource requirements for the stages to be allocated after system selection at time T is as follows: .

7. The Spark SQL task scheduling optimization method based on dynamic resources according to claim 6, characterized in that, In step S4, when At that time, the task resources required for the m stages are sorted in ascending order, and k is the minimum value that satisfies the following constraints: ， In the formula, R represents the sequence of resource requirements for each stage of the resource allocation process, and R represents the total number of remaining resources in the Spark cluster at the current time T.

8. The Spark SQL task scheduling optimization method based on dynamic resources according to claim 6, characterized in that, In step S4, the minimum number of task resources allocated is forced to be 1.

9. A Spark SQL task scheduling optimization method based on dynamic resources according to claim 8, characterized in that, In step S5, the method for normalizing and setting upper limit protection for the preliminary resource allocation of each stage of resources to be allocated is as follows: The actual resources allocated in the first k stages As shown in the following formula: ， In the formula, This represents the total number of remaining resources in the Spark cluster at the current time T; This indicates the upper limit of resource allocation set for each stage; This represents the allocatable resources for the i-th stage where resources are to be allocated.

Images