Spark task running gateway implementation method and system

Abstract

The present invention provides a Spark task running gateway implementation method and system, which relates to the field of resource scheduling technology, including receiving dual cluster configuration information; establishing connections with YARN and Kubernetes clusters; obtaining resource status information and establishing a task distribution table; receiving Spark task requests; selecting a target cluster according to a resource load index; converting the task request into a corresponding task description file; submitting the task and recording the response information. The present invention realizes dynamic scheduling and load balancing of dual cluster resources, and improves resource utilization and task processing efficiency.

Description

Technical Field

[0001] The present invention relates to resource scheduling technology, and in particular to a Spark task running gateway implementation method and system. Background Art

[0002] With the rapid development of big data technology, Spark, as an efficient distributed computing framework, has been widely used in data processing and analysis scenarios. Traditional Spark applications mainly run in the YARN cluster environment, and in recent years, containerized deployment solutions based on Kubernetes have gradually become an important choice. In actual production environments, enterprises often maintain both YARN clusters and Kubernetes clusters to meet the needs of different business scenarios.

[0003] Currently, the operation and management of Spark tasks mainly adopts a single cluster mode, that is, the tasks are fixedly deployed in a YARN cluster or a Kubernetes cluster. This method has the following technical problems:

[0004] In terms of resource scheduling, due to the lack of a unified task distribution mechanism, a single cluster may face resource shortage under high load conditions, while resources in other clusters are idle, making it impossible to achieve optimal resource utilization across clusters.

[0005] In terms of task management, existing technologies lack the ability to uniformly monitor and schedule Spark tasks in a dual-cluster environment. Operation and maintenance personnel need to manage different cluster environments separately, which increases the complexity and labor costs of operation and maintenance.

[0006] In terms of system reliability, when a cluster fails or encounters a performance bottleneck, the lack of an automated task switching mechanism makes it impossible to guarantee business continuity and stability, affecting the availability of the overall system. Summary of the invention

[0007] The embodiments of the present invention provide a Spark task running gateway implementation method and system, which can solve the problems in the prior art.

[0008] According to a first aspect of the embodiments of the present invention,

[0009] Provides the Spark task running gateway implementation method, including:

[0010] Receive dual-cluster configuration information, the dual-cluster configuration information including YARN cluster configuration parameters, Kubernetes cluster configuration parameters, gateway load threshold, resource balance coefficient and task switching threshold; establish connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; periodically obtain resource status information of the YARN cluster and the Kubernetes cluster, calculate the resource load index based on the resource status information, and establish a dual-cluster task distribution table, the dual-cluster task distribution table records the resource load index, available computing resources and task running records of each cluster;

[0011] Receive a Spark task request, wherein the Spark task request includes a task type identifier, resource requirement parameters, a task priority identifier, and a task execution timeout; obtain the current resource load index of the YARN cluster and the Kubernetes cluster from the dual-cluster task distribution table; when the resource load indexes of both clusters do not exceed the gateway load threshold, allocate the Spark task request to the cluster with the lowest resource load index among the YARN cluster and the Kubernetes cluster, and use the cluster with the lowest resource load index as the target cluster; when the resource load index of the target cluster exceeds the gateway load threshold, determine whether the resource load index of the other cluster meets the task running conditions, and if so, use the cluster that meets the task running conditions as the new target cluster;

[0012] The Spark task request is converted into a corresponding task description file according to the type of the target cluster, wherein a YARN application description file including resource requirement parameters is generated for the YARN cluster, and a Pod configuration file including container specification definitions is generated for the Kubernetes cluster; the task description file is submitted to the target cluster to obtain task submission response information, wherein the task submission response information includes a task identifier, an initial running status, and resource allocation details; and the task submission response information is recorded in the dual-cluster task distribution table.

[0013] Establishing connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; and periodically acquiring resource status information of the YARN cluster and the Kubernetes cluster includes:

[0014] Establishing a first connection with the YARN cluster according to the YARN cluster configuration parameters, establishing a second connection with the Kubernetes cluster using the Kubernetes cluster configuration parameters, and acquiring resource status information from the YARN cluster and the Kubernetes cluster respectively according to a preset collection period, wherein the resource status information includes the number of computing nodes, node CPU usage data, node memory usage data, and the number of currently running applications;

[0015] Calculating the resource load index of the YARN cluster and the Kubernetes cluster based on the resource status information, wherein the resource load index is obtained by weighted calculation, and the weighted calculation includes: multiplying the average of the node CPU usage data by a first weight coefficient, multiplying the average of the node memory usage data divided by the node maximum memory capacity by a second weight coefficient, dividing the number of currently running applications by a preset maximum number of applications and multiplying the result by a third weight coefficient, and taking the sum of the three products as the resource load index, wherein the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is 1;

[0016] A dynamic load threshold is calculated based on the difference between the resource load index of the YARN cluster and the resource load index of the Kubernetes cluster. The calculation of the dynamic load threshold includes: dividing the difference by a preset maximum load index to obtain a first ratio, multiplying an adjustment coefficient by the first ratio to obtain a second ratio, adding 1 to the second ratio and multiplying the result by a basic load threshold to obtain the dynamic load threshold, wherein the adjustment coefficient is greater than 0 and less than 1.

[0017] Calculating a resource load index based on the resource status information and establishing a dual-cluster task distribution table include:

[0018] A dual-cluster task distribution table is constructed, wherein the dual-cluster task distribution table includes a cluster identification field, a current resource load index field, an available computing resource field, and a task running record field, wherein the available computing resource field records the number of available CPU cores and the available memory capacity, and the task running record field records historical load change data, the ratio of the number of completed tasks to the total number of tasks, the ratio of the total task running time to the number of tasks, and the ratio of the actual resource usage to the allocated resource amount.

[0019] Generating a YARN application description file containing resource requirement parameters for the YARN cluster includes:

[0020] Obtain resource request parameters and resource restriction parameters of the application, wherein the resource request parameters include processor requirements, memory requirements, and storage requirements, and the resource restriction parameters include a minimum resource restriction value, a maximum resource restriction value, and a resource elasticity coefficient, and obtain the initial load characteristics of the application according to the resource request parameters;

[0021] Based on the initial load characteristics, the running status data of the application program is collected, the running status data including the task completion time limit, the resource usage intensity, and the task queue status, the task completion time limit is converted into a task urgency value, the resource usage intensity is converted into a resource density value, and the task queue status is converted into a queue impact factor value;

[0022] Calculating the dynamic priority of the application program according to the task urgency value, the resource density value, and the queue impact factor value, and multiplying the task urgency value, the resource density value, and the queue impact factor value by corresponding weight coefficients respectively and summing them to obtain the application program priority score;

[0023] Dynamically adjust the resource elasticity coefficient according to the application priority score to obtain a target elasticity coefficient, multiply the resource request parameter by the target elasticity coefficient to obtain a first resource amount, compare the first resource amount with the maximum resource limit value, select the smaller value between the two as the second resource amount, compare the second resource amount with the minimum resource limit value, and select the larger value between the two as the container target resource amount;

[0024] A YARN application description file is generated based on the container target resource amount and the application priority score, wherein the YARN application description file includes container resource configuration information and task scheduling priority information, wherein the container resource configuration information is determined according to the container target resource amount, and the task scheduling priority information is determined according to the application priority score.

[0025] The Pod configuration file containing the container specification definition generated for the Kubernetes cluster includes:

[0026] Acquire multidimensional resource requirement parameters of the container, the multidimensional resource requirement parameters including processor core requirement, memory requirement, graphics processor requirement, storage space requirement, and network bandwidth requirement, and construct the multidimensional resource requirement parameters into a resource requirement feature vector;

[0027] Collect historical load data of the container based on the resource demand feature vector, calculate the load mean of the historical load data, divide the sum of squares of the difference between the load value at the sampling moment and the load mean by the number of sampling points to obtain a basic fluctuation value, and multiply the basic fluctuation value by a time attenuation weight to obtain a load fluctuation degree;

[0028] Calculate the resource limit parameters of the container according to the load fluctuation, and determine the upper limit of the container resources; obtain the hardware attribute parameters and topological structure parameters of the node, perform feature matching on the hardware attribute parameters of the node according to the upper limit of the container resources to obtain a node matching score, and calculate the topological distance between Pods based on the topological structure parameters to obtain a dispersion score;

[0029] A Pod configuration file is generated according to the container resource upper limit, the node matching score, and the dispersion score. The Pod configuration file includes container specification definition information, node affinity rule information, and anti-affinity constraint information. The container specification definition information is determined according to the container resource upper limit, the node affinity rule information is determined according to the node matching score, and the anti-affinity constraint information is determined according to the dispersion score.

[0030] The method further comprises:

[0031] Periodically collect task running status information based on the task identifier, the task running status information including task execution progress, resource usage, and running time; when it is detected that the task running time exceeds the task execution timeout, obtain the execution log of the task in the target cluster;

[0032] Analyze the task exception type according to the execution log, and the task exception type includes resource competition exception, data skew exception, and memory overflow exception; when it is judged to be a resource competition exception, calculate the current resource load index of another cluster, and if it is lower than the task switching threshold, terminate the current task and migrate the Spark task request to another cluster for resubmission;

[0033] When it is determined to be a data skew exception or a memory overflow exception, optimize the resource configuration in the resource demand parameter, and resubmit the optimized task in the target cluster; update the task running record in the dual-cluster task distribution table, and the task running record includes task running time, resource usage, task switching times, and exception type statistics.

[0034] A second aspect of an embodiment of the present invention provides a Spark task running gateway implementation system, including:

[0035] The first unit is used to receive dual-cluster configuration information, wherein the dual-cluster configuration information includes YARN cluster configuration parameters, Kubernetes cluster configuration parameters, gateway load threshold, resource balance coefficient, and task switching threshold; establish connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; periodically obtain resource status information of the YARN cluster and the Kubernetes cluster, calculate the resource load index based on the resource status information, and establish a dual-cluster task distribution table, wherein the dual-cluster task distribution table records the resource load index, available computing resources, and task running records of each cluster;

[0036] The second unit is used to receive a Spark task request, wherein the Spark task request includes a task type identifier, resource requirement parameters, a task priority identifier, and a task execution timeout; obtain the current resource load index of the YARN cluster and the Kubernetes cluster from the dual-cluster task distribution table; when the resource load index of both clusters does not exceed the gateway load threshold, allocate the Spark task request to the cluster with the lowest resource load index among the YARN cluster and the Kubernetes cluster, and use the cluster with the lowest resource load index as the target cluster; when the resource load index of the target cluster exceeds the gateway load threshold, determine whether the resource load index of the other cluster meets the task running conditions, and if so, use the cluster that meets the task running conditions as the new target cluster;

[0037] The third unit is used to convert the Spark task request into a corresponding task description file according to the type of the target cluster, wherein a YARN application description file including resource requirement parameters is generated for the YARN cluster, and a Pod configuration file including container specification definitions is generated for the Kubernetes cluster; submit the task description file to the target cluster to obtain task submission response information, wherein the task submission response information includes a task identifier, an initial running status, and resource allocation details; and record the task submission response information in the dual-cluster task distribution table.

[0038] A third aspect of the embodiments of the present invention

[0039] An electronic device is provided, comprising:

[0040] processor;

[0041] a memory for storing processor-executable instructions;

[0042] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0043] A fourth aspect of the embodiments of the present invention is:

[0044] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0045] The beneficial effects of this application are as follows:

[0046] By establishing a dual-cluster task distribution table and periodically updating resource status information, real-time monitoring of resource usage of the YARN cluster and Kubernetes cluster is achieved, providing an accurate decision-making basis for task scheduling and improving the efficiency of cluster resource utilization.

[0047] By adopting a task distribution strategy based on the resource load index and combining the judgment mechanism of the gateway load threshold and task running conditions, the intelligent scheduling and load balancing of Spark tasks between the two clusters are realized, avoiding the overload of resources in a single cluster and ensuring the stable operation of tasks.

[0048] The task description file format is automatically converted according to different cluster types to generate configuration files that comply with the YARN cluster and Kubernetes cluster specifications, which simplifies the cross-cluster task submission process, improves task deployment efficiency, and reduces the workload of operation and maintenance personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a method for implementing a Spark task running gateway according to an embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of a Spark task running gateway implementation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0053] Figure 1The flowchart of the Spark task running gateway implementation method according to the embodiment of the present invention is as follows: Figure 1 As shown, the method includes:

[0054] S101. Receive dual-cluster configuration information, the dual-cluster configuration information including YARN cluster configuration parameters, Kubernetes cluster configuration parameters, gateway load threshold, resource balance coefficient and task switching threshold; establish connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; periodically obtain resource status information of the YARN cluster and the Kubernetes cluster, calculate the resource load index based on the resource status information, and establish a dual-cluster task distribution table, the dual-cluster task distribution table records the resource load index, available computing resources and task running records of each cluster;

[0055] S102. Receive a Spark task request, wherein the Spark task request includes a task type identifier, resource requirement parameters, a task priority identifier, and a task execution timeout; obtain the current resource load index of the YARN cluster and the Kubernetes cluster from the dual-cluster task distribution table; when the resource load index of both clusters does not exceed the gateway load threshold, allocate the Spark task request to the cluster with the lowest resource load index among the YARN cluster and the Kubernetes cluster, and use the cluster with the lowest resource load index as the target cluster; when the resource load index of the target cluster exceeds the gateway load threshold, determine whether the resource load index of the other cluster meets the task running conditions, and if so, use the cluster that meets the task running conditions as the new target cluster;

[0056] S103. Convert the Spark task request into a corresponding task description file according to the type of the target cluster, wherein a YARN application description file including resource requirement parameters is generated for the YARN cluster, and a Pod configuration file including container specification definitions is generated for the Kubernetes cluster; submit the task description file to the target cluster to obtain task submission response information, wherein the task submission response information includes a task identifier, an initial running status, and resource allocation details; and record the task submission response information in the dual-cluster task distribution table.

[0057] In an optional implementation, establishing connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; and periodically acquiring resource status information of the YARN cluster and the Kubernetes cluster includes:

[0058] Establishing a first connection with the YARN cluster according to the YARN cluster configuration parameters, establishing a second connection with the Kubernetes cluster using the Kubernetes cluster configuration parameters, and acquiring resource status information from the YARN cluster and the Kubernetes cluster respectively according to a preset collection period, wherein the resource status information includes the number of computing nodes, node CPU usage data, node memory usage data, and the number of currently running applications;

[0059] Calculating the resource load index of the YARN cluster and the Kubernetes cluster based on the resource status information, wherein the resource load index is obtained by weighted calculation, and the weighted calculation includes: multiplying the average of the node CPU usage data by a first weight coefficient, multiplying the average of the node memory usage data divided by the node maximum memory capacity by a second weight coefficient, dividing the number of currently running applications by a preset maximum number of applications and multiplying the result by a third weight coefficient, and taking the sum of the three products as the resource load index, wherein the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is 1;

[0060] A dynamic load threshold is calculated based on the difference between the resource load index of the YARN cluster and the resource load index of the Kubernetes cluster. The calculation of the dynamic load threshold includes: dividing the difference by a preset maximum load index to obtain a first ratio, multiplying an adjustment coefficient by the first ratio to obtain a second ratio, adding 1 to the second ratio and multiplying the result by a basic load threshold to obtain the dynamic load threshold, wherein the adjustment coefficient is greater than 0 and less than 1.

[0061] First, read the connection parameters of the YARN cluster and Kubernetes cluster through the configuration file. The YARN cluster configuration parameters include the ResourceManager address, port number, authentication information, etc.; the Kubernetes cluster configuration parameters include the APIServer address, authentication token, namespace, etc.

[0062] For YARN cluster connection, use the Apache Hadoop client API to create a YarnClient object, set the configuration parameters, and call the start() method to start the connection. The system will verify the correctness of the connection parameters. If the verification fails, it will record an error log and retry the connection until a stable connection is established.

[0063] For Kubernetes cluster connection, use the official Java client library to create a KubernetesClient object, configure the cluster access credentials, and establish a secure connection with the API Server. The system will also perform connection verification to ensure that cluster resources can be accessed normally.

[0064] Set the resource status information collection cycle to 60 seconds. In each collection cycle, obtain resource status data from two clusters:

[0065] Get the NodeManager list from the YARN cluster, and traverse each node to collect information such as CPU usage, memory usage, and the number of currently running containers. For example, a node has a CPU usage of 75%, a memory usage of 8 GB / total memory of 16 GB, and runs 10 containers.

[0066] Get the Node list from the Kubernetes cluster and use metrics-server to collect the resource usage of each node, including CPU usage, memory usage, number of running Pods, etc. For example, a node has a CPU usage of 60%, memory usage of 12GB / total memory of 32GB, and runs 15 Pods.

[0067] Next, calculate the resource load index of the two clusters. Set the weight coefficients: CPU usage weight 0.4, memory usage weight 0.4, and application number weight 0.2. Assuming that the average CPU usage of the YARN cluster is 80%, the average memory usage is 75%, and the number of running applications is 60% of the maximum limit, its load index is 0.736. Similarly, calculate the load index of the Kubernetes cluster.

[0068] The dynamic load threshold is calculated based on the load index difference between the two clusters. Set the maximum load index to 1.0, the adjustment coefficient to 0.8, and the basic load threshold to 0.7. If the YARN cluster load index is 0.736 and the Kubernetes cluster load index is 0.528, the load difference is 0.208. Divide the difference by the maximum load index to get 0.208, multiply it by the adjustment coefficient to get 0.1664, add 1 to get 1.1664, and the final dynamic load threshold is 0.816.

[0069] This application can achieve unified monitoring and management of heterogeneous computing clusters by establishing real-time connection between the two clusters and periodically collecting resource status, thereby improving cluster operation and maintenance efficiency. The weighted calculation method is used to comprehensively consider multi-dimensional indicators such as CPU, memory, and number of applications, making resource load assessment more comprehensive and accurate, and able to truly reflect the cluster load status. The dynamic load threshold mechanism is introduced to adaptively adjust the threshold according to the load difference between the two clusters, avoiding the limitations brought by fixed thresholds and improving the cluster load balancing effect.

[0070] In an optional implementation, calculating a resource load index based on the resource status information and establishing a dual-cluster task distribution table includes:

[0071] A dual-cluster task distribution table is constructed, wherein the dual-cluster task distribution table includes a cluster identification field, a current resource load index field, an available computing resource field, and a task running record field, wherein the available computing resource field records the number of available CPU cores and the available memory capacity, and the task running record field records historical load change data, the ratio of the number of completed tasks to the total number of tasks, the ratio of the total task running time to the number of tasks, and the ratio of the actual resource usage to the allocated resource amount.

[0072] The construction of dual-cluster task distribution table and resource load management is a complex technical process that requires multiple links to achieve accurate task scheduling and resource allocation.

[0073] First, the resource status information in the cluster is collected in real time through the resource monitoring module. Each cluster node is scanned regularly to collect basic indicators such as CPU usage, memory usage, and disk IO. The sampling interval can be set to 30 seconds to ensure the real-time nature of the data. The system stores the collected raw data in a time series database for subsequent analysis and calculation.

[0074] After obtaining the basic monitoring data, calculate the resource load index. The specific process is to normalize indicators such as CPU usage and memory occupancy to obtain a standardized value between 0 and 1. Then set the weight according to the importance of each indicator, such as CPU usage weight 0.4, memory occupancy weight 0.3, and disk IO weight 0.3. The final resource load index is obtained through weighted calculation.

[0075] Next, build a dual-cluster task distribution table, which contains the following core fields:

[0076] - Cluster ID field: used to uniquely identify different clusters in UUID format;

[0077] - Current resource load index: records the calculated load index value;

[0078] - Available computing resources: including the number of available CPU cores and memory capacity, such as "CPU: 24cores, Memory: 128GB";

[0079] - Task running records: including multiple statistical indicators;

[0080] - Historical load change data: record recent load fluctuations;

[0081] - Task completion rate: number of completed tasks / total number of tasks;

[0082] - Average task duration: total running time / number of tasks;

[0083] - Resource utilization: actual usage / allocated amount.

[0084] The system will update the data in the task distribution table regularly. For example, the current resource load index will be updated every 5 minutes, and the task running record will be updated every hour. The historical data will retain the records of the last 7 days for analyzing load trends.

[0085] In actual applications, when a new task arrives, the system first queries the task distribution table to obtain the status of the two clusters. By comparing indicators such as the current load index and available resources, the more suitable cluster is selected to execute the task. At the same time, historical task operation records are taken into consideration to avoid assigning computationally intensive tasks to clusters with high recent loads.

[0086] This application can achieve accurate load perception through real-time monitoring and dynamic updating of cluster status, avoid the problem of unbalanced resource allocation, and improve the overall computing resource utilization efficiency.

[0087] Establish a unified decision-making basis for task distribution, evaluate cluster status through multi-dimensional indicators, make task scheduling more reasonable, reduce task queue waiting time, and improve business response speed.

[0088] Completely record historical operation data, providing reliable data support for cluster expansion and resource planning. At the same time, by analyzing the historical load change patterns, it can predict future resource requirements and implement more intelligent task scheduling strategies.

[0089] In an optional implementation, generating a YARN application description file including resource requirement parameters for the YARN cluster includes:

[0090] Obtain resource request parameters and resource restriction parameters of the application, wherein the resource request parameters include processor requirements, memory requirements, and storage requirements, and the resource restriction parameters include a minimum resource restriction value, a maximum resource restriction value, and a resource elasticity coefficient, and obtain the initial load characteristics of the application according to the resource request parameters;

[0091] Based on the initial load characteristics, the running status data of the application program is collected, the running status data including the task completion time limit, the resource usage intensity, and the task queue status, the task completion time limit is converted into a task urgency value, the resource usage intensity is converted into a resource density value, and the task queue status is converted into a queue impact factor value;

[0092] Calculating the dynamic priority of the application program according to the task urgency value, the resource density value, and the queue impact factor value, and multiplying the task urgency value, the resource density value, and the queue impact factor value by corresponding weight coefficients respectively and summing them to obtain the application program priority score;

[0093] Dynamically adjust the resource elasticity coefficient according to the application priority score to obtain a target elasticity coefficient, multiply the resource request parameter by the target elasticity coefficient to obtain a first resource amount, compare the first resource amount with the maximum resource limit value, select the smaller value between the two as the second resource amount, compare the second resource amount with the minimum resource limit value, and select the larger value between the two as the container target resource amount;

[0094] A YARN application description file is generated based on the container target resource amount and the application priority score, wherein the YARN application description file includes container resource configuration information and task scheduling priority information, wherein the container resource configuration information is determined according to the container target resource amount, and the task scheduling priority information is determined according to the application priority score.

[0095] The method for generating a YARN application description file including resource requirement parameters for a YARN cluster specifically comprises the following steps:

[0096] First, obtain the resource request parameters and resource restriction parameters of the application. Resource request parameters include processor requirements, memory requirements, and storage requirements. For example, the processor requirement may be 4-core CPU, the memory requirement may be 8GB, and the storage requirement may be 100GB. Resource restriction parameters include the minimum resource restriction value, the maximum resource restriction value, and the resource elasticity coefficient. For example, the minimum resource restriction value may be 2-core CPU, 4GB memory, and 50GB storage, the maximum resource restriction value may be 8-core CPU, 16GB memory, and 200GB storage, and the initial value of the resource elasticity coefficient may be 1.2. Based on these resource request parameters, the system analyzes the initial load characteristics of the application, such as compute-intensive, memory-intensive, or storage-intensive.

[0097] Next, the running status data of the application is collected based on the initial load characteristics. The running status data includes the task completion time limit, resource usage intensity, and task queue status. For example, the task completion time limit may be 2 hours, the resource usage intensity may be 80% CPU usage and 70% memory usage, and the task queue status may be a queue length of 10. The system converts the task completion time limit into a task urgency value, such as converting 2 hours into an urgency value of 0.8 (the closer to 1, the more urgent it is). The resource usage intensity is converted into a resource density value, such as averaging the CPU and memory usage to obtain a resource density value of 0.75. The task queue status is converted into a queue impact factor value, such as converting a queue length of 10 into a queue impact factor value of 0.6 (the closer to 1, the greater the impact).

[0098] Then, the dynamic priority of the application is calculated based on the task urgency value, resource intensity value, and queue impact factor value. The system multiplies these three values ​​with the preset weight coefficients and sums them to obtain the application priority score. For example, assuming the weight coefficients are 0.4, 0.3, and 0.3 respectively, the priority score is 0.8 * 0.4 + 0.75 * 0.3 + 0.6* 0.3 = 0.725.

[0099] Next, dynamically adjust the resource elasticity coefficient according to the application priority score to obtain the target elasticity coefficient. For example, the priority score of 0.725 can be mapped to the range of 1.1 to 1.5 to obtain a target elasticity coefficient of 1.3. The system multiplies the resource request parameter by the target elasticity coefficient to obtain the first resource amount. For example, the processor requirement is 4 cores * 1.3 = 5.2 cores. Then compare the first resource amount with the maximum resource limit value, and select the smaller value of the two as the second resource amount. In this example, 5.2 cores are less than the maximum limit of 8 cores, so the second resource amount is 5.2 cores. Finally, compare the second resource amount with the minimum resource limit value, and select the larger value of the two as the container target resource amount. 5.2 cores are greater than the minimum limit of 2 cores, so the container target resource amount is 5.2 cores. Similar calculations are performed for memory and storage resources.

[0100] Finally, a YARN application description file is generated based on the container target resource amount and the application priority score. The file includes container resource configuration information and task scheduling priority information. The container resource configuration information is determined based on the container target resource amount, such as CPU: 5.2 cores, memory: 10.4GB, storage: 130GB. The task scheduling priority information is determined based on the application priority score, such as mapping 0.725 to the YARN priority level, which may correspond to the "HIGH" priority.

[0101] This application can achieve:

[0102] This method can dynamically adjust resource allocation according to the actual resource requirements and operating status of the application, improving resource utilization efficiency. By considering task urgency, resource density and queue status, more intelligent and flexible task scheduling is achieved, improving overall system performance.

[0103] This method introduces the concept of resource elasticity coefficient and dynamically adjusts it according to the application priority, which not only guarantees the resource requirements of high-priority tasks but also avoids over-allocation of resources, thus achieving balanced and optimized resource allocation.

[0104] By generating a YARN application description file containing detailed resource configuration and priority information, a more accurate scheduling basis is provided for the YARN scheduler, which helps to improve the overall operation efficiency and resource utilization of the cluster.

[0105] In an optional implementation, generating a Pod configuration file containing a container specification definition for the Kubernetes cluster includes:

[0106] Acquire multidimensional resource requirement parameters of the container, the multidimensional resource requirement parameters including processor core requirement, memory requirement, graphics processor requirement, storage space requirement, and network bandwidth requirement, and construct the multidimensional resource requirement parameters into a resource requirement feature vector;

[0107] Collect historical load data of the container based on the resource demand feature vector, calculate the load mean of the historical load data, divide the sum of squares of the difference between the load value at the sampling moment and the load mean by the number of sampling points to obtain a basic fluctuation value, and multiply the basic fluctuation value by a time attenuation weight to obtain a load fluctuation degree;

[0108] Calculate the resource limit parameters of the container according to the load fluctuation, and determine the upper limit of the container resources; obtain the hardware attribute parameters and topological structure parameters of the node, perform feature matching on the hardware attribute parameters of the node according to the upper limit of the container resources to obtain a node matching score, and calculate the topological distance between Pods based on the topological structure parameters to obtain a dispersion score;

[0109] A Pod configuration file is generated according to the container resource upper limit, the node matching score, and the dispersion score. The Pod configuration file includes container specification definition information, node affinity rule information, and anti-affinity constraint information. The container specification definition information is determined according to the container resource upper limit, the node affinity rule information is determined according to the node matching score, and the anti-affinity constraint information is determined according to the dispersion score.

[0110] First, for the collection of resource demand characteristics of containers, the system will collect data in the following dimensions:

[0111] CPU core usage: collect CPU usage per second through cAdvisor; memory usage: collect memory usage data when the container is running; GPU usage: collect GPU device utilization data;

[0112] Storage space: monitors the disk read and write volume of the container; Network bandwidth: collects the container network packet sending and receiving data;

[0113] Based on the collected historical load data, the system will perform the following processing:

[0114] 1. For each resource dimension, obtain the sampling data every minute in the last 7 days;

[0115] 2. Calculate the average usage of each resource as the benchmark load;

[0116] 3. Count the deviation between the actual load and the average value to get the fluctuation situation;

[0117] 4. Introducing the time decay factor, recent data has a higher weight;

[0118] For example, if the average CPU usage of a container in the past 7 days is 2 cores and the standard deviation is 0.5 cores, the system will set the CPU limit to 3 cores (the average plus 1.5 times the standard deviation).

[0119] Next, the system will obtain the hardware parameters of the cluster nodes:

[0120] - CPU model and number of cores;

[0121] - Memory capacity and type;

[0122] - GPU model and video memory size;

[0123] - Disk type and capacity;

[0124] - Network card bandwidth specifications;

[0125] Based on these parameters, the system calculates how well the node matches the container:

[0126] - Hardware specification matching: Check whether the node meets the resource requirements of the container;

[0127] - Load balancing: evaluates the resource balance of the node after adding the container;

[0128] - Affinity score: Calculate the optimal deployment location based on the existing container distribution;

[0129] The resulting Pod configuration file contains:

[0130] 1. The specific value of the container resource limit;

[0131] 2. Node selectors and affinity rules;

[0132] 3. Pod anti-affinity settings to ensure distributed deployment of containers.

[0133] This application enables:

[0134] Through accurate resource demand analysis and dynamic load calculation, container resource configuration is made more reasonable, resource waste is avoided, and cluster resource utilization is improved. Intelligent scheduling based on node hardware characteristics and topology structure achieves optimal distribution of containers and improves application performance and availability. The configuration file is automatically generated in a data-driven way to reduce the complexity and error probability of manual configuration and improve operation and maintenance efficiency.

[0135] In an optional implementation, the method further includes:

[0136] Periodically collect task running status information based on the task identifier, the task running status information including task execution progress, resource usage, and running time; when it is detected that the task running time exceeds the task execution timeout, obtain the execution log of the task in the target cluster;

[0137] Analyze the task exception type according to the execution log, and the task exception type includes resource competition exception, data skew exception, and memory overflow exception; when it is judged to be a resource competition exception, calculate the current resource load index of another cluster, and if it is lower than the task switching threshold, terminate the current task and migrate the Spark task request to another cluster for resubmission;

[0138] When it is determined to be a data skew exception or a memory overflow exception, optimize the resource configuration in the resource demand parameter, and resubmit the optimized task in the target cluster; update the task running record in the dual-cluster task distribution table, and the task running record includes task running time, resource usage, task switching times, and exception type statistics.

[0139] The status monitoring implementation based on task identifiers is executed in a scheduled task mode. The monitoring program obtains task status information from the cluster through the task identifier every 30 seconds, including indicators such as the current execution progress percentage, CPU usage, memory usage, and disk IO status. For example, for the task identifier "TASK-2024-001", specific values ​​such as progress 80%, CPU usage 85%, and memory usage 12GB can be obtained.

[0140] When it is detected that the task running time exceeds the preset timeout period (such as 2 hours), the system automatically obtains the execution log of the task from the target cluster. Log acquisition uses the cluster log interface to extract the running log information of the last 30 minutes, including abnormal information of ERROR and WARN levels.

[0141] The exception analysis module identifies the exception type based on the preset feature rules. Resource competition exceptions are manifested as a large amount of "Resource temporarily unavailable" information; data skew exceptions are manifested in that the processing time of some task shards is much longer than that of other shards; and memory overflow exceptions are manifested in "OutOfMemoryError" related errors.

[0142] When resource contention anomalies are identified, the system calculates the resource load of the candidate cluster. The calculation method is based on the weighted results of indicators such as CPU usage, memory usage, and disk IO. If the resource load index of the candidate cluster is lower than the set threshold (such as 60%), task migration is performed: first stop the current task, save the task running status and configuration information, and then resubmit the task to the candidate cluster.

[0143] For data skew anomalies, the system optimizes the configuration by increasing the number of data shards (for example, from 100 to 200) and adjusting the data sampling rate (to 1%). For memory overflow anomalies, the executor memory configuration is appropriately increased (for example, from 4GB to 6GB), and the garbage collection strategy is adjusted. The optimized configuration is used for task resubmission.

[0144] The task running records are updated to the dual-cluster task distribution table, which contains detailed information such as actual running time, average resource usage, number of task switching between clusters, number of various exceptions, etc. These records are used for subsequent task scheduling optimization.

[0145] This application can achieve:

[0146] By periodically collecting task status and analyzing abnormal causes, real-time monitoring and timely intervention of Spark task running status are achieved, improving the stability and reliability of task running.

[0147] Based on multi-level exception handling strategies, including task migration, parameter optimization and other means, a complete task exception handling mechanism was established, which effectively reduced the task failure rate and improved the utilization efficiency of computing resources.

[0148] By recording task operation data in detail, we have accumulated rich historical information on task execution, provided data support for subsequent task scheduling optimization, and achieved continuous improvement in task execution efficiency.

[0149] Figure 2FIG. 1 is a schematic diagram of the structure of the Spark task running gateway implementation system according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0150] The first unit is used to receive dual-cluster configuration information, wherein the dual-cluster configuration information includes YARN cluster configuration parameters, Kubernetes cluster configuration parameters, gateway load threshold, resource balance coefficient, and task switching threshold; establish connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; periodically obtain resource status information of the YARN cluster and the Kubernetes cluster, calculate the resource load index based on the resource status information, and establish a dual-cluster task distribution table, wherein the dual-cluster task distribution table records the resource load index, available computing resources, and task running records of each cluster;

[0151] The second unit is used to receive a Spark task request, wherein the Spark task request includes a task type identifier, resource requirement parameters, a task priority identifier, and a task execution timeout; obtain the current resource load index of the YARN cluster and the Kubernetes cluster from the dual-cluster task distribution table; when the resource load index of both clusters does not exceed the gateway load threshold, allocate the Spark task request to the cluster with the lowest resource load index among the YARN cluster and the Kubernetes cluster, and use the cluster with the lowest resource load index as the target cluster; when the resource load index of the target cluster exceeds the gateway load threshold, determine whether the resource load index of the other cluster meets the task running conditions, and if so, use the cluster that meets the task running conditions as the new target cluster;

[0152] The third unit is used to convert the Spark task request into a corresponding task description file according to the type of the target cluster, wherein a YARN application description file including resource requirement parameters is generated for the YARN cluster, and a Pod configuration file including container specification definitions is generated for the Kubernetes cluster; submit the task description file to the target cluster to obtain task submission response information, wherein the task submission response information includes a task identifier, an initial running status, and resource allocation details; and record the task submission response information in the dual-cluster task distribution table.

[0153] According to a third aspect of the embodiments of the present invention,

[0154] An electronic device is provided, comprising:

[0155] processor;

[0156] a memory for storing processor-executable instructions;

[0157] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0158] According to a fourth aspect of the embodiments of the present invention,

[0159] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0160] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for implementing a Spark task running gateway, characterized in that: include: Receive dual-cluster configuration information, the dual-cluster configuration information including YARN cluster configuration parameters, Kubernetes cluster configuration parameters, gateway load threshold, resource balance coefficient and task switching threshold; establish connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; periodically obtain resource status information of the YARN cluster and the Kubernetes cluster, calculate the resource load index based on the resource status information, and establish a dual-cluster task distribution table, the dual-cluster task distribution table records the resource load index, available computing resources and task running records of each cluster; Receive a Spark task request, wherein the Spark task request includes a task type identifier, resource requirement parameters, a task priority identifier, and a task execution timeout; obtain the current resource load index of the YARN cluster and the Kubernetes cluster from the dual-cluster task distribution table; when the resource load indexes of both clusters do not exceed the gateway load threshold, distribute the Spark task request to the cluster with the lowest resource load index among the YARN cluster and the Kubernetes cluster, and use the cluster with the lowest resource load index as the target cluster; When the resource load index of the target cluster exceeds the gateway load threshold, determining whether the resource load index of another cluster meets the task running condition, and if so, taking the cluster that meets the task running condition as the new target cluster; Convert the Spark task request into a corresponding task description file according to the type of the target cluster, wherein a YARN application description file including resource requirement parameters is generated for the YARN cluster, and a Pod configuration file including container specification definitions is generated for the Kubernetes cluster; submit the task description file to the target cluster, and obtain task submission response information, wherein the task submission response information includes a task identifier, an initial running status, and resource allocation details; The task submission response information is recorded in the dual-cluster task distribution table.

2. The method according to claim 1, characterized in that Establishing connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; and periodically acquiring resource status information of the YARN cluster and the Kubernetes cluster includes: Establishing a first connection with the YARN cluster according to the YARN cluster configuration parameters, establishing a second connection with the Kubernetes cluster using the Kubernetes cluster configuration parameters, and acquiring resource status information from the YARN cluster and the Kubernetes cluster respectively according to a preset collection period, wherein the resource status information includes the number of computing nodes, node CPU usage data, node memory usage data, and the number of currently running applications; Calculating the resource load index of the YARN cluster and the Kubernetes cluster based on the resource status information, wherein the resource load index is obtained by weighted calculation, and the weighted calculation includes: multiplying the average of the node CPU usage data by a first weight coefficient, multiplying the average of the node memory usage data divided by the node maximum memory capacity by a second weight coefficient, dividing the number of currently running applications by a preset maximum number of applications and multiplying the result by a third weight coefficient, and taking the sum of the three products as the resource load index, wherein the sum of the first weight coefficient, the second weight coefficient and the third weight coefficient is 1; A dynamic load threshold is calculated based on the difference between the resource load index of the YARN cluster and the resource load index of the Kubernetes cluster. The calculation of the dynamic load threshold includes: dividing the difference by a preset maximum load index to obtain a first ratio, multiplying an adjustment coefficient by the first ratio to obtain a second ratio, adding 1 to the second ratio and multiplying the result by a basic load threshold to obtain the dynamic load threshold, wherein the adjustment coefficient is greater than 0 and less than 1.

3. The method according to claim 2, characterized in that Calculating a resource load index based on the resource status information and establishing a dual-cluster task distribution table include: A dual-cluster task distribution table is constructed, wherein the dual-cluster task distribution table includes a cluster identification field, a current resource load index field, an available computing resource field, and a task running record field, wherein the available computing resource field records the number of available CPU cores and the available memory capacity, and the task running record field records historical load change data, the ratio of the number of completed tasks to the total number of tasks, the ratio of the total task running time to the number of tasks, and the ratio of the actual resource usage to the allocated resource amount.

4. The method according to claim 1, characterized in that: Generating a YARN application description file containing resource requirement parameters for the YARN cluster includes: Obtain resource request parameters and resource restriction parameters of the application, wherein the resource request parameters include processor requirements, memory requirements, and storage requirements, and the resource restriction parameters include a minimum resource restriction value, a maximum resource restriction value, and a resource elasticity coefficient, and obtain the initial load characteristics of the application according to the resource request parameters; Based on the initial load characteristics, the running status data of the application program is collected, the running status data including the task completion time limit, the resource usage intensity, and the task queue status, the task completion time limit is converted into a task urgency value, the resource usage intensity is converted into a resource density value, and the task queue status is converted into a queue impact factor value; Calculating the dynamic priority of the application program according to the task urgency value, the resource density value, and the queue impact factor value, and multiplying the task urgency value, the resource density value, and the queue impact factor value by corresponding weight coefficients respectively and summing them to obtain the application program priority score; Dynamically adjust the resource elasticity coefficient according to the application priority score to obtain a target elasticity coefficient, multiply the resource request parameter by the target elasticity coefficient to obtain a first resource amount, compare the first resource amount with the maximum resource limit value, select the smaller value between the two as the second resource amount, compare the second resource amount with the minimum resource limit value, and select the larger value between the two as the container target resource amount; A YARN application description file is generated based on the container target resource amount and the application priority score, wherein the YARN application description file includes container resource configuration information and task scheduling priority information, wherein the container resource configuration information is determined according to the container target resource amount, and the task scheduling priority information is determined according to the application priority score.

5. The method according to claim 1, characterized in that: The Pod configuration file containing the container specification definition generated for the Kubernetes cluster includes: Acquire multidimensional resource requirement parameters of the container, the multidimensional resource requirement parameters including processor core requirement, memory requirement, graphics processor requirement, storage space requirement, and network bandwidth requirement, and construct the multidimensional resource requirement parameters into a resource requirement feature vector; Collect historical load data of the container based on the resource demand feature vector, calculate the load mean of the historical load data, divide the sum of squares of the difference between the load value at the sampling moment and the load mean by the number of sampling points to obtain a basic fluctuation value, and multiply the basic fluctuation value by a time attenuation weight to obtain a load fluctuation degree; Calculate the resource limit parameters of the container according to the load fluctuation, and determine the upper limit of the container resources; obtain the hardware attribute parameters and topological structure parameters of the node, perform feature matching on the hardware attribute parameters of the node according to the upper limit of the container resources to obtain a node matching score, and calculate the topological distance between Pods based on the topological structure parameters to obtain a dispersion score; A Pod configuration file is generated according to the container resource upper limit, the node matching score, and the dispersion score. The Pod configuration file includes container specification definition information, node affinity rule information, and anti-affinity constraint information. The container specification definition information is determined according to the container resource upper limit, the node affinity rule information is determined according to the node matching score, and the anti-affinity constraint information is determined according to the dispersion score.

6. The method according to claim 1, characterized in that The method further comprises: Periodically collect task running status information based on the task identifier, the task running status information including task execution progress, resource usage, and running time; when it is detected that the task running time exceeds the task execution timeout, obtain the execution log of the task in the target cluster; Analyze the task exception type according to the execution log, and the task exception type includes resource competition exception, data skew exception, and memory overflow exception; when it is judged to be a resource competition exception, calculate the current resource load index of another cluster, and if it is lower than the task switching threshold, terminate the current task and migrate the Spark task request to another cluster for resubmission; When it is determined to be a data skew exception or a memory overflow exception, optimize the resource configuration in the resource demand parameter, and resubmit the optimized task in the target cluster; update the task running record in the dual-cluster task distribution table, and the task running record includes task running time, resource usage, task switching times, and exception type statistics.

7. A Spark task running gateway implementation system, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The first unit is used to receive dual-cluster configuration information, wherein the dual-cluster configuration information includes YARN cluster configuration parameters, Kubernetes cluster configuration parameters, gateway load threshold, resource balance coefficient, and task switching threshold; establish connections with the YARN cluster and the Kubernetes cluster respectively according to the YARN cluster configuration parameters and the Kubernetes cluster configuration parameters; periodically obtain resource status information of the YARN cluster and the Kubernetes cluster, calculate the resource load index based on the resource status information, and establish a dual-cluster task distribution table, wherein the dual-cluster task distribution table records the resource load index, available computing resources, and task running records of each cluster; The second unit is used to receive a Spark task request, wherein the Spark task request includes a task type identifier, a resource requirement parameter, a task priority identifier, and a task execution timeout; obtain the current resource load index of the YARN cluster and the Kubernetes cluster from the dual-cluster task distribution table; when the resource load indexes of the two clusters do not exceed the gateway load threshold, allocate the Spark task request to the cluster with the lowest resource load index among the YARN cluster and the Kubernetes cluster, and use the cluster with the lowest resource load index as the target cluster; When the resource load index of the target cluster exceeds the gateway load threshold, determining whether the resource load index of another cluster meets the task running condition, and if so, taking the cluster that meets the task running condition as the new target cluster; The third unit is used to convert the Spark task request into a corresponding task description file according to the type of the target cluster, wherein a YARN application description file including resource requirement parameters is generated for the YARN cluster, and a Pod configuration file including container specification definitions is generated for the Kubernetes cluster; submit the task description file to the target cluster, and obtain task submission response information, wherein the task submission response information includes a task identifier, an initial running status, and resource allocation details; The task submission response information is recorded in the dual-cluster task distribution table.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images