Spark data locality optimization system in Kubernetes environment

Abstract

A SPARK data locality optimization system in a KUBERNETES environment comprises a Spark driving module and a Kubernetes network plug-in module, the Spark driving module calls a system instruction to obtain a network address IP of an HDFS and cluster related information according to a calculation task request of a client, calculates data locality according to the network address IP information and then performs node distribution on a calculation task; and the Kubernetes network plug-in module receives calling of the Spark driving module and specifies an IP network address of a container where the calculation task is located for the calculation task. According to the method, the data locality of the Spark in the Kubernetes environment is improved by optimizing the Spark driving module and adjusting the network plug-ins in the Kubernetes, the network transmission throughput overhead in the Spark task execution process can be remarkably reduced while recognition of the Spark operation on the HDFS data IP address can be achieved, and then the operation execution performance of the Spark in the Kubernetes environment is optimized.

Description

Technical Field

[0001] The present invention relates to a technology in the field of distributed data processing, specifically a SPARK data locality optimization system in a KUBERNETES environment. Background Art

[0002] With the rapid development of big data technology, Apache Spark has become a leading platform for large-scale data processing. Its in-memory computing mechanism and easy-to-use API provide unprecedented speed for processing huge data sets. At the same time, Kubernetes, as a widely adopted container orchestration system, has also become an ideal environment for running and managing Spark jobs due to its efficient resource management capabilities. However, when deploying Spark jobs to Kubernetes, the industry faces a series of technical challenges that affect the performance and efficiency of data processing tasks. For example, how to optimize data locality and reduce inter-cluster network transmission. When Spark is deployed in a distributed cluster for computing, different levels of data locality are defined for a single computing task, including: PROCESS_LOCAL (process local), NODE_LOCAL (node ​​local), NO_PREF (no preference), RACK_LOCAL (rack local) and ANY (anywhere). In the default configuration, Spark will try to maximize the use of higher levels of data locality to minimize network transmission overhead. Specifically, when data locality is PROCESS_LOCAL and NODE_LOCAL, the data and computing tasks are located in the same node and can be computed without network transmission, while the remaining data locality levels require data transmission across nodes, resulting in low execution efficiency. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a SPARK data locality optimization system in a Kubernetes environment. By optimizing the Spark driver module and adjusting the network plug-in in Kubernetes, the data locality of Spark in the Kubernetes environment is improved, which can realize the recognition of the HDFS data IP address by the Spark job and significantly reduce the network transmission throughput overhead during the execution of the Spark task, thereby optimizing the job execution performance of Spark in the Kubernetes environment.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a SPARK data locality optimization system in a KUBERNETES environment, comprising: a Spark driver module and a Kubernetes network plug-in module, wherein: the Spark driver module calls a system instruction to obtain a network address IP and cluster related information of a HDFS according to a computing task request of a client, calculates data locality according to the network address IP information, and then allocates nodes to the computing task; the Kubernetes network plug-in module receives the call of the Spark driver module and specifies the IP network address of the container where the computing task is located.

[0006] The Spark driver module includes: an initialization unit, a computing resource judgment unit, a data locality processing unit and a task scheduling integration unit, wherein: the initialization unit initializes the task configuration data according to the task startup information of the client, the computing resource judgment unit judges whether the execution requirements are met according to the task configuration data, HDFS data and HDFS node information, the data locality processing unit intervenes in the computing resource allocation and task placement according to the HDFS data, and the task scheduling integration unit integrates the information and generates a YAML configuration file and submits it to the Kubernetes network plug-in module.

[0007] The task configuration data refers to: according to the Spark task configuration of the client, the HDFS network address IP is set as the ENV environment variable in all nodes, and after the HDFS command hdfs dfsadmin-report is executed through the HDFS network address IP to obtain the HDFS node information of all nodes in the current cluster, the system call top command is called to obtain the maximum number of CPU cores C and the maximum available memory M of the workload node in the current cluster, and these data are output to the computing resource judgment unit and the data locality processing unit respectively.

[0008] The Spark task configuration includes the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, and the HDFS network address IP.

[0009] The HDFS node information includes: the actual data distribution in HDFS (the node address of the data backup distribution required), and the network information of each node (the actual IP address).

[0010] The satisfying execution requirements means: after extracting the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, the maximum number of CPU cores C and the maximum available memory M of each workload node from the task configuration data, if it is determined that c>=C or m>=M or c*e>=C*N or m*e>M*N, a system error is prompted and the task exits; otherwise, execution continues, where N is the workload node.

[0011] The intervention mentioned above refers to: labeling the nodes in the cluster according to the distribution of HDFS data, that is, when there are M nodes (M < N) among the total N workload nodes {node1, node2... node N} on which HDFS actually places data, adding the compute affinity label compute_label = true to these M nodes in Kubernetes, and keeping the remaining N - M nodes unchanged.

[0012] The submission mentioned above refers to: integrating all the information output by the initialization module into a YAML configuration file and using the command kubectl apply -f cfg.yaml to push it to the Kubernetes network plugin module through the HDFS network address IP.

[0013] The YAML configuration file includes: the number of Spark executors e, the number of CPU cores c and the memory size m required for a single Spark executor, and the HDFS network address IP.

[0014] The specification of the IP network address of the container where the computing task is located refers to: according to the network creation request of the Spark driver module, the Kubernetes network plugin module designates the network address IP of the working container as the same network address IP as the target node of the working container, that is, updating the hostnetwork configuration of Kubernetes to true. Technical effects

[0015] The present invention is implemented through a newly developed Spark driver module and an adjusted Kubernetes network plugin. This method allows Spark jobs to effectively identify and utilize the network addresses of HDFS data, thereby significantly improving data locality and reducing the need for cross-node data transmission. Compared with the prior art, by optimizing data locality processing, the present invention not only reduces the overhead of network transmission, but also improves the efficiency of job execution and resource utilization. This improvement enables Spark to run better in the modern container management platform of Kubernetes, enhances its task processing efficiency in cloud platforms such as Kubernetes, and reduces performance waste caused by network overhead. Brief description of the drawings

[0016] Figure 1 It is a schematic diagram of the system of the present invention;

[0017] Figure 2 It is a flowchart of the method of the present invention;

[0018] Figure 3A bar chart comparing data locality experiments before and after using the data locality optimization system;

[0019] Figure 4 A bar chart comparing the performance of various algorithms before and after using the data locality optimization system. DETAILED DESCRIPTION

[0020] like Figure 1 As shown, a SPARK data locality optimization system in a KUBERNETES environment involved in this embodiment includes: a Spark driver module and a Kubernetes network plug-in module including an initialization unit, a computing resource judgment unit, a data locality processing unit and a task scheduling integration unit, wherein: the client submits a request to the Spark driver module and sends it to the initialization unit; the initialization unit obtains HDFS data information and cluster resource information respectively according to the specific information of the request, sends the task configuration data node information to the computing resource judgment unit, sends the HDFS data distribution to the data locality processing unit, and sends all the above information to the task scheduling integration unit; the computing resource judgment unit determines whether the task can be executed in the cluster according to the task configuration information and cluster resource information; the data locality processing unit intervenes in task allocation using affinity tags for HDFS data distribution; the task scheduling integration unit integrates all information into a YAML configuration file and submits it to Kubernetes, and sends the HDFS network address IP to the Kubernetes network plug-in module; The Kubernetes network plugin module modifies the network address of the working container in Kubernetes.

[0021] The client may be any Spark client, does not require local deployment or remote deployment, and may also be a benchmark test suite.

[0022] The Kubernetes described is a Kubernetes system pre-deployed in the cluster, using a default unmodified version that includes a working container.

[0023] The working container is a container used for actual computing, which is isolated from the external environment. There can be several working containers, which can be on any working node.

[0024] The Kubernetes network plugin module described uses the Kubernetes interface and is not part of the pre-deployed Kubernetes system.

[0025] like Figure 2 As shown, the locality optimization method based on the above system in this embodiment includes:

[0026] Step 1: According to the task request of the client, configure the number of Spark executors e, the number of CPU cores c and the memory size m required for a single Spark executor, and the HDFS network address IP.

[0027] Step 2: Initialize the task: Obtain HDFS node information, obtain the maximum resources of the cluster, and set the ENV environment variables for all nodes, specifically including:

[0028] 2.1 Obtain HDFS node information, specifically by executing the HDFS instruction hdfs dfsadmin - report to obtain the HDFS node information of all nodes in the current cluster, including: the actual data distribution in HDFS, that is, the node addresses where the data backups to be used are distributed, and the network information of each node, that is, the actual IP address.

[0029] 2.2 Call the top instruction to obtain the maximum number of CPU cores C and the maximum available memory M of the workload nodes in the current cluster.

[0030] 2.3 Set the actual IP address as the ENV environment variable among all nodes.

[0031] Step 3: Determine whether the cluster resources meet the execution requirements, specifically: when c >= C or m >= M or c * e >= C * N or m * e > M * N, prompt a system error and exit; otherwise, execute Step 4.

[0032] Step 4: Label affinity labels for the workload nodes where the data is actually stored: There are N workload nodes {node1, node2…node N}, among which M nodes (M < N) have data actually placed in HDFS. Then, for these M nodes, set the compute affinity label compute_label = true in Kubernetes, and the remaining N - M nodes remain unchanged.

[0033] Step 5: Integrate the YAML configuration file: Integrate all the information mentioned above into a complete configuration file that can be read by Kubernetes.

[0034] Step 6: Submit the YAML configuration file to Kubernetes and modify the container network configuration options, specifically including:

[0035] 6.1 Use the instruction kubectl apply - f cfg.yaml to submit the YAML configuration file to Kubernetes.

[0036] 6.2 Modify the container network configuration options related to Kubernetes, that is, update the hostnetwork configuration to true.

[0037] The YAML configuration file is in the configuration file format required by Kubernetes and is used to initiate a request to the existing Kubernetes environment.

[0038] After specific actual experiments, under the specific cluster configuration in Table 1, the system was benchmarked using the Hibench component as the access client. Here, the Hibench component is an open source big data computing performance test suite. There are many algorithms for Spark to test. Here, ten commonly used algorithms are selected as test algorithms, including: DFSIOE, Repartition, Sleep, Sort, TeraSort, WordCount, KMeans, PageRank, Bayes, and Join. In the test, the Gigantic data scale (1TB) under the default settings of Hibench was selected for testing.

[0039] Table 1 Test cluster configuration information table

[0040] The data locality and computation time data of each algorithm before and after the system is optimized using data locality under Hibench as the benchmark suite in this embodiment are shown in Table 2.

[0041] Table 2 Data locality and computation time of each algorithm before and after using data locality optimization system under Hibench as benchmark suite

[0042] In the table, ANY indicates a task with poor data locality, that is, the computing task cannot perceive the data source; NODE_LOCAL indicates a task with good data locality, that is, the computing task and the data are on the same node.

[0043] like Figure 3 The following is a bar chart of the data locality part in Table 2. It records the performance of different algorithms before and after the application of the data locality optimization module under the Gigantic data scale (1TB) by analyzing the experimental log. It can be clearly seen from the figure that before the data locality optimization module was used, the number of ANY jobs in all the algorithms involved in the test was significantly higher than the number of NODE_LOCAL. After the data locality optimization measures were introduced, the number of NODE_LOCAL jobs increased significantly, even reaching the same level as the number of ANY jobs before the optimization.

[0044] like Figure 4As shown in the figure, it is a bar chart of the time efficiency part in Table 2, showing the overall execution time changes of various algorithms before and after data locality optimization under the Gigantic data scale (1TB). The observation results show that the execution time of most algorithms has been reduced. It is worth mentioning that the KMeans algorithm has achieved significant results after optimization, with the execution time shortened from about 39 minutes to about 32 minutes, and an overall execution time reduction of nearly 18%. This result fully demonstrates that data locality optimization can bring significant performance improvements in some algorithms.

[0045] Compared with the prior art, the present invention optimizes data locality. Compared with the system that does not use data locality optimization, for specific algorithms (such as more complex algorithms such as KMeans), it can significantly improve data locality performance, increase the number of cases where data and calculations are on the same node, and achieve an overall time performance improvement of about 15%.

[0046] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A SPARK data locality optimization system in a KUBERNETES environment, characterized in that: include: Spark driver module and Kubernetes network plug-in module, where: the Spark driver module calls system instructions to obtain the network address IP and cluster-related information of HDFS according to the client's computing task request, and calculates the data locality according to the network address IP information, and then allocates nodes to the computing task; the Kubernetes network plug-in module receives the call of the Spark driver module and specifies the IP network address of the container where the computing task is located.

2. The SPARK data locality optimization system in the KUBERNETES environment according to claim 1, characterized in that: The Spark driver module includes: an initialization unit, a computing resource judgment unit, a data locality processing unit and a task scheduling integration unit, wherein: the initialization unit initializes the task configuration data according to the task startup information of the client, the computing resource judgment unit judges whether the execution requirements are met according to the task configuration data, HDFS data and HDFS node information, the data locality processing unit intervenes in the computing resource allocation and task placement according to the HDFS data, and the task scheduling integration unit integrates the information and generates a YAML configuration file and submits it to the Kubernetes network plug-in module.

3. The SPARK data locality optimization system in the KUBERNETES environment according to claim 2, characterized in that: The task configuration data refers to: according to the Spark task configuration of the client, the HDFS network address IP is set as the ENV environment variable in all nodes, and after the HDFS command hdfs dfsadmin-report is executed through the HDFS network address IP to obtain the HDFS node information of all nodes in the current cluster, the system call top command is called to obtain the maximum number of CPU cores C and the maximum available memory M of the workload node in the current cluster, and these data are output to the computing resource judgment unit and the data locality processing unit respectively.

4. The SPARK data locality optimization system in the KUBERNETES environment according to claim 2, characterized in that: The Spark task configuration includes the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, and the HDFS network address IP.

5. The SPARK data locality optimization system in the KUBERNETES environment according to claim 2, characterized in that: The HDFS node information includes: the actual data distribution in HDFS (the node address of the data backup distribution required), and the network information of each node (the actual IP address).

6. The SPARK data locality optimization system in the KUBERNETES environment according to claim 2, characterized in that: The satisfying execution requirements means: after extracting the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, the maximum number of CPU cores C and the maximum available memory M of each workload node from the task configuration data, if it is determined that c>=C or m>=M or c*e>=C*N or m*e>M*N, a system error is prompted and the task exits; otherwise, execution continues, where N is the workload node.

7. The SPARK data locality optimization system in a KUBERNETES environment according to claim 2, characterized in that: The intervention mentioned above refers to: labeling the nodes in the cluster according to the distribution of HDFS data. That is, when there are M (M < N) nodes among the total N workload nodes {node1, node2…node N} where the actual HDFS placement has data, these M nodes are labeled with the compute affinity label compute_label = true in Kubernetes, and the remaining N - M nodes remain unchanged.

8. The SPARK data locality optimization system in a KUBERNETES environment according to claim 2, characterized in that: The submission means: integrating all the information output by the initialization module into a YAML configuration file and using the command kubectlapply-f cfg.yaml to push it to the Kubernetes network plug-in module through the HDFS network address IP; The YAML configuration file includes: the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, and the HDFS network address IP; The aforementioned specification of the IP network address of the container where the computing task is located means that the Kubernetes network plug-in module specifies the network address IP of the working container as the same network address IP as the target node of the working container according to the network creation request of the Spark driver module, that is, updates the hostnetwork configuration of Kubernetes to true.

9. A local optimization method based on the system according to any one of claims 1 to 8, characterized in that: include: Step 1: According to the client's task request, configure the number of Spark executors e, the number of CPU cores c and memory size m required for a single Spark executor, and the HDFS network address IP; Step 2: Initialization tasks: Get HDFS node information, get the maximum cluster resources, and set ENV environment variables for all nodes, including: 2.1 Obtain HDFS node information. Specifically, execute the HDFS command hdfs dfsadmin-report to obtain the HDFS node information of all nodes in the current cluster, including: the actual data distribution in HDFS, that is, the node address of the data backup distribution required, and the network information of each node, that is, the actual IP address; 2.2 Call the top command to obtain the maximum number of CPU cores C and the maximum available memory M of the workload node in the current cluster; 2.3 Set the actual IP address as the ENV environment variable in all nodes; Step 3: Determine whether the cluster resources meet the execution requirements. Specifically: when c>=C or m>=M or c*e>=C*N or m*e>M*N, a system error is displayed and the system exits. Otherwise, execute step 4. Step 4: Label the affinity label for the workload nodes that actually store data. There are N workload nodes {node1, node2... node N}, where M of them (M < N) are the nodes where data is actually placed in HDFS. Then, set the compute affinity label compute_label = true for these M nodes in Kubernetes, and leave the remaining N - M nodes unchanged; Step 5: Integrate the YAML configuration file: Integrate all the above information into a complete configuration file that can be read by Kubernetes; Step 6: Submit the YAML configuration file to Kubernetes and modify the container network configuration options, including: 6.1 Use the command kubectl apply-f cfg.yaml to submit the YAML configuration file to Kubernetes; 6.2 Modify the Kubernetes configuration options for container networks, that is, update the hostnetwork configuration to true.

Images