Spark data locality optimization system in kubernetes environment

Abstract

A SPARK data locality optimization system in a KUBERNETES environment comprises a Spark driver module and a Kubernetes network plug-in module, wherein: the Spark driver module, according to a computing task request of a client, calls a system instruction to obtain a network address IP of HDFS and cluster related information and calculates data locality according to the network address IP information and then performs node allocation on the computing task; the Kubernetes network plug-in module receives the call of the Spark driver module and specifies the IP network address of the container where the computing task is located. The application improves the data locality of Spark in the Kubernetes environment by optimizing the Spark driver module and adjusting the network plug-in in the Kubernetes, can realize the identification of the HDFS data IP address by the Spark job, significantly reduces the network transmission throughput cost in the Spark task execution process, and further optimizes the job execution performance of Spark in the Kubernetes environment.

Description

Technical Field

[0001] This invention relates to a technology in the field of distributed data processing, specifically a Spark data locality optimization system in the KUBERNETES environment. Background Technology

[0002] With the rapid development of big data technology, Apache Spark has become a leading platform for large-scale data processing. Its in-memory computing mechanism and easy-to-use API provide unprecedented speed for processing massive datasets. Meanwhile, Kubernetes, as a widely adopted container orchestration system, is an ideal environment for running and managing Spark jobs due to its efficient resource management capabilities. However, when deploying Spark jobs to Kubernetes, the industry faces a series of technical challenges that affect the performance and efficiency of data processing tasks. For example, how to optimize data locality and reduce inter-cluster network transmission. When Spark is deployed in a distributed cluster for computation, different levels of data locality are defined for individual computation tasks, including: PROCESS_LOCAL (process-local), NODE_LOCAL (node-local), NO_PREF (no preference), RACK_LOCAL (rack-local), and ANY (any location). By default, Spark attempts to maximize the use of higher levels of data locality to minimize network transmission overhead. Specifically, when the data locality is PROCESS_LOCAL and NODE_LOCAL, the data and computation tasks reside on the same node and can be computed without network transmission. However, the remaining data locality levels require data transmission across nodes, resulting in lower execution efficiency. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a Spark data locality optimization system for the Kubernetes environment. By optimizing the Spark driver module and adjusting the network plugin within Kubernetes, the system improves Spark's data locality in the Kubernetes environment. This enables Spark jobs to identify HDFS data IP addresses while significantly reducing network transmission throughput overhead during Spark task execution, thereby optimizing Spark job execution performance in the Kubernetes environment.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a Spark data locality optimization system in a Kubernetes environment, comprising: a Spark driver module and a Kubernetes network plugin module, wherein: the Spark driver module, based on the client's computing task request, calls system instructions to obtain the HDFS network address IP and cluster-related information, calculates data locality based on the network address IP information, and then allocates nodes for the computing task; the Kubernetes network plugin module receives the call from the Spark driver module and specifies the IP network address of the container where the computing task resides.

[0006] The Spark driver module includes an initialization unit, a computing resource judgment unit, a data locality processing unit, and a task scheduling integration unit. The initialization unit initializes the task configuration data based on the client's task startup information. The computing resource judgment unit determines whether the execution requirements are met based on the task configuration data, HDFS data, and HDFS node information. The data locality processing unit intervenes in the allocation of computing resources and task placement based on HDFS data. The task scheduling integration unit integrates the information, generates a YAML configuration file, and submits it to the Kubernetes network plugin module.

[0007] The task configuration data refers to: based on the client's Spark task configuration, setting the HDFS network address IP as an ENV environment variable on all nodes, and then executing the HDFS command hdfs dfsadmin-report through the HDFS network address IP to obtain the HDFS node information of all nodes in the current cluster. After that, the system call top command is invoked to obtain the maximum number of CPU cores C and the maximum usable memory M of the workload nodes in the current cluster, and these data are output to the computing resource judgment unit and the data locality processing unit respectively.

[0008] The Spark task configuration includes the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, and the HDFS network address IP.

[0009] The HDFS node information includes: the actual data distribution in HDFS (the node addresses of the data backup distribution needed) and the network information (actual IP address) of each node.

[0010] The aforementioned meeting of execution requirements means: after extracting the number of Spark executors e, the number of CPU cores c and memory size m required by a single Spark executor, the maximum number of CPU cores C and the maximum usable memory M for each workload node from the task configuration data, and determining whether c>=C or m>=M or c*e>=C*N or m*e>M*N, if so, prompting a system error and exiting, otherwise continuing execution, where N is the workload node.

[0011] The intervention mentioned above refers to: labeling the nodes in the cluster according to the distribution of HDFS data. That is, when there are M nodes with actual HDFS data placement among a total of N workload nodes {node1, node2... node N}, and M < N, the compute affinity label compute_label = true is added to these M nodes in Kubernetes, while the remaining N - M nodes remain unchanged.

[0012] The submission mentioned above refers to: integrating all the information output by the initialization module into a YAML configuration file and using the command kubectl apply -f cfg.yaml to push it to the Kubernetes network plugin module through the HDFS network address IP.

[0013] The YAML configuration file includes: the number of Spark executors e, the number of CPU cores c and the memory size m required for a single Spark executor, and the HDFS network address IP.

[0014] The specification of the IP network address of the container where the computing task is located refers to: the Kubernetes network plugin module, according to the network creation request of the Spark driver module, specifies the network address IP of the working container as the same network address IP as the target node of the working container, that is, updates the hostnetwork configuration of Kubernetes to true.

[0015] Technical effects

[0016] The present invention is implemented through a newly developed Spark driver module and an adjusted Kubernetes network plugin. This method allows Spark jobs to effectively identify and utilize the network addresses of HDFS data, thereby significantly improving data locality and reducing the need for cross-node data transmission. Compared with the prior art, the present invention, by optimizing data locality processing, not only reduces the overhead of network transmission, but also improves the efficiency of job execution and resource utilization. This improvement enables Spark to better run in the modern container management platform of Kubernetes, enhances its task processing efficiency in the cloud platform of Kubernetes, and reduces the performance waste caused by network overhead. Brief description of the drawings

[0017] Figure 1 It is a schematic diagram of the system of the present invention;

[0018] Figure 2 It is a flowchart of the method of the present invention;

[0019] Figure 3A bar chart comparing data locality experiments before and after system optimization using data locality;

[0020] Figure 4 A bar chart comparing the performance of various algorithms before and after optimizing the system using data locality. Detailed Implementation

[0021] like Figure 1 As shown in this embodiment, a Spark data locality optimization system in a Kubernetes environment includes: a Spark driver module and a Kubernetes network plugin module comprising an initialization unit, a computing resource judgment unit, a data locality processing unit, and a task scheduling integration unit. Specifically: the client submits a request to the Spark driver module, which is then sent to the initialization unit; the initialization unit, based on the specific information in the request, obtains HDFS data information and cluster resource information, sends the task configuration data node information to the computing resource judgment unit, sends the HDFS data distribution to the data locality processing unit, and sends all the above information to the task scheduling integration unit; the computing resource judgment unit determines whether the task can be executed in the cluster based on the task configuration information and cluster resource information; the data locality processing unit intervenes in task allocation using affinity tags for the HDFS data distribution; and the task scheduling integration unit integrates all information into a YAML configuration file, submits it to Kubernetes, and sends the HDFS network address (IP) to the Kubernetes network plugin module.

[0022] The Kubernetes network plugin module allows you to modify the network address of a working container in Kubernetes.

[0023] The client can be any Spark client, and local or remote deployment is not required. It can also be a benchmark suite.

[0024] The Kubernetes system mentioned is a Kubernetes system pre-deployed in the cluster, using the default unmodified version, which includes worker containers.

[0025] The aforementioned working container is the container actually used for computation, isolated from the external environment, and there can be several of them, which can be located on any working node.

[0026] The Kubernetes network plugin module mentioned above uses the Kubernetes interface and is not part of a pre-deployed Kubernetes system.

[0027] like Figure 2 As shown, this embodiment illustrates a locality optimization method based on the aforementioned system, including:

[0028] Step 1: According to the task request of the client, configure the number of Spark executors e, the number of CPU cores c and the memory size m required for a single Spark executor, as well as the HDFS network address IP.

[0029] Step 2: Initialize the task: Obtain HDFS node information, obtain the maximum resources of the cluster, and set the ENV environment variables for all nodes, specifically including:

[0030] 2.1 Obtain HDFS node information. Specifically, execute the HDFS instruction hdfs dfsadmin - report to obtain the HDFS node information of all nodes in the current cluster, including: the actual data distribution in HDFS, that is, the node addresses of the data backup distribution required, and the network information of each node, that is, the actual IP address.

[0031] 2.2 Invoke the top instruction to obtain the maximum number of CPU cores C and the maximum available memory M of the workload nodes in the current cluster.

[0032] 2.3 Set the actual IP address as the ENV environment variable in all nodes.

[0033] Step 3: Determine whether the cluster resources meet the execution requirements. Specifically: When c >= C or m >= M or c * e >= C * N or m * e > M * N, prompt a system error and exit; otherwise, execute Step 4.

[0034] Step 4: Label affinity labels for the workload nodes where the data is actually stored: There are N workload nodes {node1, node2…node N}, among which M nodes (M < N) actually have data placed in HDFS. Then, set the compute affinity label compute_label = true for these M nodes in Kubernetes, and leave the remaining N - M nodes unchanged.

[0035] Step 5: Integrate the YAML configuration file: Integrate all the information mentioned above into a complete configuration file that can be read by Kubernetes.

[0036] Step 6: Submit the YAML configuration file to Kubernetes and modify the container network configuration options, specifically including:

[0037] 6.1 Use the instruction kubectl apply - f cfg.yaml to submit the YAML configuration file to Kubernetes.

[0038] 6.2 Modify the container network configuration options related to Kubernetes, that is, update the hostnetwork configuration to true.

[0039] The YAML configuration file is in the configuration file format required by Kubernetes and is used to make requests to the existing Kubernetes environment.

[0040] Through specific practical experiments, under the specific cluster configuration in Table 1, the Hibench component was used as the access client to conduct benchmark tests on the system. The Hibench component is an open-source big data computing performance testing suite, offering various algorithms for Spark testing. Ten commonly used algorithms were selected for testing: DFSIOE, Repartition, Sleep, Sort, TeraSort, WordCount, KMeans, PageRank, Bayes, and Join. The test used a Gigantic data size (1TB) under the default Hibench settings.

[0041] Table 1 Test Cluster Configuration Information

[0042]

[0043] Table 2 shows the data locality and computation time data of each algorithm before and after using Hibench as the benchmark suite to optimize the system based on data locality.

[0044] Table 2 shows the data locality and computation time of each algorithm before and after using the data locality optimization system with Hibench as the benchmark suite.

[0045]

[0046] In the table, ANY represents tasks with poor data locality, meaning the computation task is unaware of the data source; NODE_LOCAL represents tasks with good data locality, meaning the computation task and the data reside on the same node.

[0047] like Figure 3 The bar chart shown in Table 2 illustrates the data locality aspect, detailing the performance of different algorithms before and after applying the data locality optimization module on a Gigantic dataset of 1TB, based on a detailed analysis of the experimental logs. The chart clearly shows that before using the data locality optimization module, the number of ANY jobs was significantly higher than the number of NODE_LOCAL jobs in all tested algorithms. After introducing data locality optimization, the number of NODE_LOCAL jobs significantly increased, even reaching a level equal to the number of ANY jobs before optimization.

[0048] like Figure 4As shown in Table 2, the bar chart for time efficiency illustrates the overall execution time changes of various algorithms before and after data locality optimization for a Gigantic data size of 1TB. The results show that the execution time of most algorithms decreased. Notably, the KMeans algorithm achieved a significant improvement after optimization, reducing its execution time from approximately 39 minutes to approximately 32 minutes, representing an overall reduction of nearly 18%. This result clearly demonstrates that data locality optimization can bring significant performance improvements to certain algorithms.

[0049] Compared with existing technologies, this invention optimizes data locality. Compared with systems that do not use data locality optimization, for specific algorithms (such as more complex algorithms like KMeans), it can significantly improve data locality performance, increase the number of cases where data and computation are on the same node, and achieve an overall time performance improvement of about 15%.

[0050] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A Spark data locality optimization system in a Kubernetes environment, characterized in that, include: The Spark driver module and the Kubernetes network plugin module are as follows: The Spark driver module, based on the client's computing task request, calls system instructions to obtain the HDFS network address (IP) and cluster-related information, calculates data locality based on the network address (IP) information, and then allocates nodes to the computing task; The Kubernetes network plugin module receives the call from the Spark driver module and specifies the IP network address of the container where the computing task is located. The Spark driver module includes an initialization unit, a computing resource judgment unit, a data locality processing unit, and a task scheduling integration unit. The initialization unit initializes task configuration data based on the client's task information. The computing resource judgment unit determines whether the execution requirements are met based on the task configuration data, HDFS data, and HDFS node information. The data locality processing unit intervenes in the allocation of computing resources and task placement based on HDFS data. The task scheduling integration unit integrates information, generates a YAML configuration file, and submits it to the Kubernetes network plugin module. The task configuration data refers to: based on the client's Spark task configuration, setting the HDFS network address IP as an ENV environment variable on all nodes, and then executing the HDFS command hdfs dfsadmin -report through the HDFS network address IP to obtain the HDFS node information of all nodes in the current cluster. After that, the system call top command is called to obtain the maximum number of CPU cores C and the maximum usable memory M of the workload nodes in the current cluster, and these data are output to the computing resource judgment unit and the data locality processing unit respectively. The intervention refers to labeling the nodes in the cluster according to the distribution of HDFS data, i.e., a total of N workload nodes. When M nodes in HDFS actually contain data, the M nodes are labeled with compute_label=true in Kubernetes, while the remaining N - M nodes remain unchanged, where M < N. The submission refers to: integrating all the information output by the initialization unit, the computing resource judgment unit, and the data locality processing unit into a YAML configuration file and pushing it to the Kubernetes network plugin module via the HDFS network address IP using the command kubectl apply -f cfg.yaml; The YAML configuration file includes: the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, and the HDFS network address IP; The aforementioned specifying the IP network address of the container where the computing task resides means that the Kubernetes network plugin module, based on the network creation request from the Spark driver module, specifies the network address IP of the working container as the same network address IP as the target node of the working container, that is, updates the Kubernetes hostnetwork configuration to true.

2. The Spark data locality optimization system in the KUBERNETES environment according to claim 1, characterized in that, The Spark task configuration includes the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, and the HDFS network address IP.

3. The Spark data locality optimization system in the KUBERNETES environment according to claim 1, characterized in that, The HDFS node information includes: the actual data distribution in HDFS, i.e., the node addresses of the data backup distribution needed, and the network information of each node, i.e., the actual IP address.

4. The Spark data locality optimization system in the KUBERNETES environment according to claim 1, characterized in that, The aforementioned meeting of execution requirements refers to: after extracting the number of Spark executors e, the number of CPU cores c and memory size m required by a single Spark executor, the maximum number of CPU cores C and the maximum usable memory M for each workload node from the task configuration data, determining whether c >= C or m >= M or c * e >= C * N or m * e > M * N, then prompting a system error and exiting; otherwise, continuing execution, where N is the workload node.

5. A locality optimization method based on the system described in any one of claims 1-4, characterized in that, include: Step 1: Based on the client's task request, configure the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, and the HDFS network address IP. Step 2, Initialization Tasks: Obtain HDFS node information, obtain the maximum cluster resources, and set the ENV environment variable for all nodes. Specifically, this includes: 2.1 Obtain HDFS node information. Specifically, obtain the HDFS node information of all nodes in the current cluster by executing the HDFS command hdfs dfsadmin -report. This includes: the actual data distribution in HDFS, i.e., the node addresses of the data backup distribution to be used, and the network information of each node, i.e., the actual IP address. 2.2 Use the `top` command to obtain the maximum number of CPU cores (C) and the maximum available memory (M) of the workload nodes in the current cluster; 2.3 Set the actual IP address as the ENV environment variable on all nodes; Step 3: Determine if the cluster resources meet the execution requirements. Specifically: if c >= C or m >= M or c * e >= C * N or m * e > M * N, then prompt a system error and exit; otherwise, proceed to step 4. Step 4: Label the workload nodes that actually store the data with affinity tags: There are N workload nodes. M nodes actually contain data in HDFS. In Kubernetes, these M nodes are labeled with compute_label=true, while the remaining N - M nodes remain unchanged, where M < N. Step 5: Integrate the YAML configuration file: Integrate all the information mentioned above into a complete configuration file that can be read by Kubernetes; Step 6: Commit the YAML configuration file to Kubernetes and modify the container network configuration options, specifically including: 6.1 Use the command `kubectl apply -f cfg.yaml` to submit the YAML configuration file to Kubernetes; 6.2 Modify the Kubernetes configuration options related to container networking, specifically by updating the hostnetwork configuration to true.

Images