A column calculation optimization method based on Spark SQL

Abstract

The present invention discloses a column calculation optimization method based on Spark SQL, comprising the following steps: S1, unified memory management, establishing a unified data management mechanism based on Arrow, after file data is loaded from disk into memory Arrow structure, it can be accessed and calculated by multiple plug-ins; S2, unified scheduling of heterogeneous computing resources, extending optimizers and plug-ins in Spark SQL, and realizing heterogeneous resource scheduling mechanism based on data features; S3, rule matching of Spark SQL logical execution plan, generating physical execution plan of CPU and GPU hybrid arrangement based on unified memory structure Arrow. The present invention compresses memory space in the format of Arrow column, avoids JVM memory calculation GC overhead, speeds up calculation efficiency, mixes and arranges Spark SQL operators into optimal execution plan according to cost optimization method, and reduces overall calculation time consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of cluster computing systems, and in particular to a column computing optimization method based on Spark SQL. Background Art

[0002] Apache Spark is a fast, general-purpose cluster computing system. It provides high-level APIs for Java, Scala, Python, and R. It also supports a rich set of high-level tools, including the SparkSQL module for SQL and structured data processing. Figure 1 shown.

[0003] Spark SQL is a SQL engine based on the open source parallel computing framework Spark, which provides the ability to query and analyze data based on the SQL language in a big data environment.

[0004] a) Spark SQL can well express the SQL data model by processing data based on RDD. Spark SQL provides Row-based data processing and analysis capabilities.

[0005] b) Spark SQL performs scheduling and processing based on Spark's parallel computing model. It can perform parallel computing and analysis based on massive data, providing high-performance data analysis and processing capabilities;

[0006] c) Spark SQL optimizes performance based on JVM through the "WholeStageCodeGen" technology, providing high-performance RowBased performance optimization.

[0007] OAP project: Provides ColumnBatch analysis and computing capabilities through the Spark SQL plug-in mechanism.

[0008] a) OAP performs column-based data management based on Apache Arrow (hereinafter referred to as Arrow). Arrow provides CPU-Cache optimized memory data management capabilities and a SIMD-friendly data management model.

[0009] b) OAP provides an LLVM based expression engine that can be combined with Arrow data to better utilize the efficient computing capabilities of CPU SIMD;

[0010] c) OAP provides Spark SQL with SIMD-optimized high-performance column computing capabilities through in-memory data optimization and expression optimization, which can significantly improve the column computing and analysis performance of Spark SQL.

[0011] Spark-RAPIDS project: a GPU-accelerated SQL engine built on the NVIDIA RAPIDS ecosystem.

[0012] a) In terms of data memory management, Arrow is used for memory layout management;

[0013] b) Provides a GPU-oriented DataFrame library, providing computing capabilities such as JOIN and aggregation operations;

[0014] c) Based on Spark's plug-in mechanism, it provides Spark with GPU optimization capabilities.

[0015] Although both OAP and Spark-RAPIDS use Arrow for memory data management, Arrow is only used as a plug-in for memory management in both projects, and data cannot be shared between the two plug-ins, that is, data cannot be shared between the CPU and GPU based on a unified data format. If both plug-ins are deployed at the same time in production, the data of the same table will need to be loaded repeatedly in both plug-ins, wasting memory resources.

[0016] Since the files are loaded into Arrow within the plug-in, data files are loaded repeatedly, resulting in a waste of I / O resources and increasing the performance pressure of the HDFS system.

[0017] At the same time, the current Spark architecture cannot provide heterogeneous unified scheduling based on data characteristics, which is mainly reflected in the following points:

[0018] Since OAP and Spark-RAPIDS are plug-in mechanisms, Spark SQL does not provide a mechanism for optimizing scheduling between plug-ins. It is not possible to use both CPU optimization and GPU optimization capabilities in one business.

[0019] In actual business projects, different fields in a table have different data features. Some fields are more suitable for CPU calculation, while some fields are more efficient for GPU calculation. For example, in the personnel information management table, statistical analysis based on numeric fields such as age, gender, and online time is more cost-effective based on CPU SIMD optimization. Retrieval and analysis of image features based on GPU acceleration can provide a better business experience. Summary of the invention

[0020] The purpose of the present invention is to provide a unified scheduling mechanism for heterogeneous computing resources based on "block" data modeling, and to realize a unified scheduling mechanism for heterogeneous computing resources based on "one piece of data". The present invention provides the following technical solutions:

[0021] The present invention provides a column calculation optimization method based on Spark SQL, comprising the following steps:

[0022] S1. Unified memory management: Establish a unified data management mechanism based on Arrow. After file data is loaded from disk into the memory Arrow structure, it can be accessed and calculated by multiple plug-ins.

[0023] S2. Unified scheduling of heterogeneous computing resources. Expand the optimizer and plug-in in Spark SQL to implement a heterogeneous resource scheduling mechanism based on data features. The specific steps include:

[0024] S2-1. Scheduling optimization mechanism based on table and field data features: Numerical calculations are preferentially scheduled to the CPU, and characteristic vector data and long string calculations are preferentially scheduled to the GPU;

[0025] S2-2. Scheduling optimization based on computing characteristics: For tasks with large data volumes and low single computing complexity, they are scheduled to the CPU first; for tasks with small data sets but complex computing tasks, they are scheduled to the GPU first;

[0026] S2-3. Implement load-based optimized scheduling: Implement scheduling optimization based on the current load of the CPU and GPU, and prioritize scheduling calculations to hardware with lower loads;

[0027] S3: Match the rules of Spark SQL's logical execution plan to generate a physical execution plan based on the unified memory structure Arrow with CPU and GPU hybrid orchestration.

[0028] As a preferred technical solution of the present invention, the Arrow structure in S1 serves as a carrier of RDD memory, which can realize memory data sharing among multiple tasks and multiple plug-ins.

[0029] As a preferred technical solution of the present invention, the native logical execution plan of Spark SQL in S3 is adapted to the physical execution plan rewritten in C++ language through rule matching. The execution plan supports the implementation of two computing devices and can implement physical execution plans of CPU and GPU types at the same time.

[0030] As a preferred technical solution of the present invention, in S3, when selecting and matching the physical execution plan node, a cost evaluation method is introduced, and the cost evaluation method includes: forming an execution plan logic of CPU and GPU hybrid arrangement based on index data and statistical data evaluation method, and the memory data format of the CPU and GPU execution plans is Arrow.

[0031] As a preferred technical solution of the present invention, in the execution plan generation stage in S3, according to the data type, data distribution, data volume, etc. that need to be calculated, it is decided whether to use the CPU or GPU operator for calculation based on the cost evaluation method. The mixed arrangement of operators of different computing devices can provide computing performance to a greater extent.

[0032] As a preferred technical solution of the present invention, in the S3, during the running phase of the execution plan, the physical execution plan is scheduled to a CPU or GPU computing device according to different types of the physical execution plan.

[0033] The beneficial effects of the present invention are:

[0034] 1. Optimize memory data structure, reduce JVM GC overhead, redefine Spark's memory computing data structure, compress memory space in the format of Arrow columns, avoid the overhead caused by JVM memory computing GC, and speed up computing efficiency;

[0035] 2. Use SIMD instructions to improve CPU operator computing efficiency. Use SIMD instruction sets to perform vector computing, so that the CPU registers can obtain more computing data at one time, accelerating the computing speed in the analysis scenario.

[0036] 3. The execution plan extension of heterogeneous computing devices reduces the overall computing time. By extending the ApacheSpark native interface, the expansion of the Spark SQL operator CPU and GPU is realized, and the Spark SQL operator is mixed and arranged into the optimal execution plan according to the cost optimization method, which reduces the overall computing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 It is a schematic diagram of the SparkSQL architecture in the prior art;

[0039] Figure 2 It is a schematic diagram of execution plan arrangement of a column calculation optimization method based on Spark SQL of the present invention. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0041] Embodiment: The present invention provides a column calculation optimization method based on Spark SQL, comprising the following steps:

[0042] S1. Unified memory management provides a unified data management mechanism based on Arrow. After file data is loaded from disk into the memory Arrow structure, it can be accessed and calculated by multiple plug-ins. Operations such as shuffle of calculation results are also implemented based on Arrow. Arrow is used as the carrier of RDD memory to realize memory data sharing among multiple tasks and multiple plug-ins.

[0043] S2. Unified scheduling of heterogeneous computing resources. Expand the optimizer and plug-in in Spark SQL to implement a heterogeneous resource scheduling mechanism based on data features. The specific steps include:

[0044] S2-1. Scheduling optimization mechanism based on table and field data features: Numerical calculations are preferentially scheduled to the CPU, and characteristic vector data and long string calculations are preferentially scheduled to the GPU;

[0045] S2-2. Scheduling optimization combined with computing features: For tasks with large data volumes and low single computing complexity, they are scheduled to the CPU first; for tasks with small data sets but complex computing tasks (such as feature comparison), they are scheduled to the GPU first;

[0046] S2-3. Implement load-based optimized scheduling: Implement scheduling optimization based on the current load of the CPU and GPU, and prioritize scheduling calculations to hardware with lower loads;

[0047] S3, matches the rules of Spark SQL's logical execution plan to generate a physical execution plan based on the unified memory structure Arrow with CPU and GPU hybrid orchestration. Figure 2 As shown in the figure, the LogicalPlan on the left is the logical execution plan of native Spark. The optimization process is mainly based on the introduction of optimization rules based on the Spark rule interface. The optimization rules will match and replace the rules of each logical execution plan in the native LogicalPlan. The replacement of the execution plan mainly depends on the cost evaluation method, which includes index information and related data statistics. According to the cost evaluation method, it is determined whether the node execution selects the CPU type operator or the GPU type operator, and finally forms a physical execution plan of CPU and GPU hybrid arrangement, and submits it to the execution engine for calculation.

[0048] Among them, Spark SQL's native logical execution plan (LogicalPlan) is adapted to the physical execution plan (PhysicalPlan) rewritten in C++ through rule matching. This execution plan supports the implementation of two computing devices, that is, it implements physical execution plans of both CPU and GPU types. Considering that the overall execution plan needs to be optimal, cost evaluation methods are introduced when selecting and matching physical execution plan nodes, including: based on index data and statistical data evaluation methods, and finally forming the execution plan logic of CPU and GPU hybrid arrangement, and the memory data format of the CPU and GPU execution plans is Arrow.

[0049] Among them, the orchestration and scheduling of heterogeneous execution plans means that the physical execution plans generated by the same SQL statement (such as FilterExec, HashAggregateExec, etc.) are implemented on both the CPU (implemented in C++ language) and the GPU. In the execution plan generation stage, the cost evaluation method is used to decide whether to use CPU or GPU operators for calculation based on the data type, data distribution, data volume, etc. that need to be calculated. The mixed orchestration of operators of different computing devices can provide computing performance to a greater extent. At the same time, in the running stage of the execution plan, the physical execution plan is scheduled to the CPU or GPU computing device according to its different types.

[0050] Among them, based on a unified memory data format, the format conversion of data on different computing devices is avoided, and a unified interface is provided for CPU operators and GPU operators; at the same time, the memory copy and serialization overhead during calculations between operators of the same type is reduced.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A column calculation optimization method based on Spark SQL, characterized in that: The steps include: S1. Unified memory management: Establish a unified data management mechanism based on Arrow. After file data is loaded from disk into the memory Arrow structure, it can be accessed and calculated by multiple plug-ins. S2. Unified scheduling of heterogeneous computing resources. Expand the optimizer and plug-in in Spark SQL to implement a heterogeneous resource scheduling mechanism based on data features. The specific steps include: S2-1. Scheduling optimization mechanism based on table and field data features: Numerical calculations are preferentially scheduled to the CPU, and characteristic vector data and long string calculations are preferentially scheduled to the GPU; S2-2. Scheduling optimization based on computing characteristics: For tasks with large data volumes and low single computing complexity, they are scheduled to the CPU first; for tasks with small data sets but complex computing tasks, they are scheduled to the GPU first; S2-3. Implement load-based optimized scheduling: Implement scheduling optimization based on the current load of the CPU and GPU, and prioritize scheduling calculations to hardware with lower loads; S3: Match the rules of Spark SQL's logical execution plan to generate a physical execution plan based on the unified memory structure Arrow with CPU and GPU hybrid orchestration.

2. The method for optimizing column calculation based on Spark SQL according to claim 1, characterized in that: The Arrow structure in S1 serves as a carrier of RDD memory, which can realize memory data sharing among multiple tasks and multiple plug-ins.

3. The method for optimizing column calculation based on Spark SQL according to claim 1, characterized in that: The native logical execution plan of Spark SQL in S3 is adapted to the physical execution plan rewritten in C++ language through rule matching. The execution plan supports the implementation of two computing devices and can implement physical execution plans of CPU and GPU types at the same time.

4. The method for optimizing column calculation based on Spark SQL according to claim 1, characterized in that: In the S3, when the physical execution plan node is selected and matched, a cost evaluation method is introduced, and the cost evaluation method includes: forming an execution plan logic of CPU and GPU hybrid arrangement based on index data and statistical data evaluation method, and the memory data format of the CPU and GPU execution plans is Arrow.

5. The method for optimizing column calculation based on Spark SQL according to claim 4, characterized in that: In the execution plan generation stage of S3, the cost evaluation method is used to determine whether to use CPU or GPU operators for calculation according to the data type, data distribution, and data volume to be calculated. The mixed arrangement of operators of different computing devices can provide computing performance to a greater extent.

6. The method for optimizing column calculation based on Spark SQL according to claim 4, characterized in that: In the S3, during the running phase of the execution plan, the physical execution plan is scheduled to a CPU or GPU computing device according to different types of the physical execution plan.

Images