Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of big data, the method comprises the following steps: in response to a received data query task, decomposing the data query task to obtain an execution plan corresponding to the data query task, the execution plan comprises a plurality of execution processes and a target operator corresponding to each execution process; converting each target operator into a target vectorization operator, distributing each target vectorization operator to a corresponding target computing node, obtaining CPU architecture information corresponding to the target computing node, and determining a target vectorization operator library matched with the CPU architecture information; whether the target vectorization operator relates to migration data reading or not is judged, the data processing process corresponding to the execution plan is executed through the target vectorization operator library according to the judgment result, and the execution result corresponding to the execution plan is obtained, so that the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, a data processing device, an electronic device and a computer-readable storage medium. Background Art

[0002] With the development of big data technology, vectorized computing technology has significantly improved the efficiency of computing tasks by processing multiple data elements simultaneously. Among them, vectorized technology is widely used in scientific computing, image processing, machine learning, and big data processing. For example, in machine learning, vectorized computing accelerates the matrix operations of neural networks; in big data processing frameworks such as Apache Spark, vectorized technology improves the efficiency of data reading and querying. However, in clusters with mixed deployment of heterogeneous CPUs, it is impossible to achieve corresponding data processing through vectorized technology, which greatly limits the application of vectorized computing in clusters with mixed deployment of heterogeneous CPUs, resulting in the inability to effectively improve the efficiency of data processing. Summary of the invention

[0003] The embodiments of the present invention provide a data processing method, device, electronic device and computer-readable storage medium to solve or partially solve the problem that vectorized computing has application limitations in heterogeneous CPU mixed deployment clusters, resulting in the inability to effectively improve data processing efficiency.

[0004] An embodiment of the present invention discloses a data processing method, which is applied to a heterogeneous CPU hybrid deployment cluster, wherein the heterogeneous CPU hybrid deployment cluster includes a plurality of servers, each of which includes a plurality of computing nodes, and the method includes:

[0005] In response to receiving a data query task, decomposing the data query task to obtain an execution plan corresponding to the data query task, wherein the execution plan includes a plurality of execution processes and a target operator corresponding to each of the execution processes;

[0006] Convert each of the target operators into a target vectorization operator, and assign each of the target vectorization operators to a corresponding target computing node, and obtain CPU architecture information corresponding to the target computing node, and determine a target vectorization operator library that matches the CPU architecture information;

[0007] It is determined whether the target vectorization operator involves reading migration data, and according to the determination result, a data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain an execution result corresponding to the execution plan.

[0008] In some feasible implementations, converting each of the target operators into a target vectorization operator includes:

[0009] Obtaining a vectorized operator dictionary for the target operator, wherein the vectorized operator dictionary maintains vectorized operators supported by the computing node;

[0010] If the target operator is located in the vectorized operator dictionary and the CPU architectures in the computing nodes all support the target operator, the target operator is converted into a target vectorized operator.

[0011] Some possible implementations also include:

[0012] If the target operator is not located in the vectorized operator dictionary, and / or the CPU architecture in the computing node does not support the target operator, a native computing engine is used to execute the execution plan.

[0013] In some feasible implementations, determining a target vectorization operator library that matches the CPU architecture information includes:

[0014] If the CPU architecture information is an x86 architecture, determining a first vectorization operator library corresponding to the x86 architecture;

[0015] If the CPU architecture information is an ARM architecture, a second vector operator library corresponding to the ARM architecture is determined.

[0016] In some feasible implementations, the target computing node includes a Native engine, and executing the data processing process corresponding to the execution plan through the target vectorized operator library according to the judgment result to obtain the execution result corresponding to the execution plan includes:

[0017] If the judgment result indicates that the target vectorization operator involves reading migration data, first CPU architecture information corresponding to an upstream computing node in the target computing node and second CPU architecture information corresponding to a downstream computing node are obtained, data conversion is performed according to the first CPU architecture information and the second CPU architecture information, and based on the result after the data conversion, a data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain an execution result corresponding to the execution plan;

[0018] If the judgment result indicates that the target vectorization operator does not involve reading migration data, the target vectorization operator is unloaded to the Native engine of the target computing node, and the underlying vectorization instruction set is called through the vectorization operator library to execute the target vectorization operator to obtain the execution result corresponding to the execution plan.

[0019] In some feasible implementations, performing data conversion according to the first CPU architecture information and the second CPU architecture information, and executing a data processing process corresponding to the execution plan through the target vectorized operator library based on a result of the data conversion to obtain an execution result corresponding to the execution plan includes:

[0020] If the first CPU architecture information is different from the second CPU architecture information, when the downstream computing node pulls the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node, and the downstream vectorized data structure is vectorized by the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0021] In some feasible implementations, performing data conversion according to the first CPU architecture information and the second CPU architecture information, and executing a data processing process corresponding to the execution plan through the target vectorized operator library based on a result of the data conversion to obtain an execution result corresponding to the execution plan includes:

[0022] If the first CPU architecture information is the same as the second CPU architecture information, the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized and calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0023] The embodiment of the present invention further discloses a data processing device, which is applied to a heterogeneous CPU hybrid deployment cluster, wherein the heterogeneous CPU hybrid deployment cluster includes a plurality of servers, each of which includes a plurality of computing nodes, and the device includes:

[0024] A task decomposition module, configured to, in response to receiving a data query task, decompose the data query task to obtain an execution plan corresponding to the data query task, wherein the execution plan includes a plurality of execution processes and a target operator corresponding to each of the execution processes;

[0025] A configuration module, used to convert each of the target operators into a target vectorization operator, and assign each of the target vectorization operators to a corresponding target computing node, and obtain CPU architecture information corresponding to the target computing node, and determine a target vectorization operator library that matches the CPU architecture information;

[0026] An execution module is used to determine whether the target vectorization operator involves reading migration data, and execute the data processing process corresponding to the execution plan through the target vectorization operator library according to the determination result to obtain the execution result corresponding to the execution plan.

[0027] In some feasible implementations, the configuration module is specifically used to:

[0028] Obtaining a vectorized operator dictionary for the target operator, wherein the vectorized operator dictionary maintains vectorized operators supported by the computing node;

[0029] If the target operator is located in the vectorized operator dictionary and the CPU architectures in the computing nodes all support the target operator, the target operator is converted into a target vectorized operator.

[0030] Some possible implementations also include:

[0031] A computing execution module is used to use a native computing engine to execute the execution plan if the target operator is not located in the vectorized operator dictionary and / or the CPU architecture in the computing node does not support the target operator.

[0032] In some feasible implementations, the configuration module is specifically used to:

[0033] If the CPU architecture information is an x86 architecture, determining a first vectorization operator library corresponding to the x86 architecture;

[0034] If the CPU architecture information is an ARM architecture, a second vector operator library corresponding to the ARM architecture is determined.

[0035] In some feasible implementations, the target computing node includes a Native engine, and the execution module is specifically used to:

[0036] If the judgment result indicates that the target vectorization operator involves reading migration data, first CPU architecture information corresponding to an upstream computing node in the target computing node and second CPU architecture information corresponding to a downstream computing node are obtained, data conversion is performed according to the first CPU architecture information and the second CPU architecture information, and based on the result after the data conversion, a data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain an execution result corresponding to the execution plan;

[0037] If the judgment result indicates that the target vectorization operator does not involve reading migration data, the target vectorization operator is unloaded to the Native engine of the target computing node, and the underlying vectorization instruction set is called through the vectorization operator library to execute the target vectorization operator to obtain the execution result corresponding to the execution plan.

[0038] In some feasible implementations, the execution module is specifically used to:

[0039] If the first CPU architecture information is different from the second CPU architecture information, when the downstream computing node pulls the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node, and the downstream vectorized data structure is vectorized by the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0040] In some feasible implementations, the execution module is specifically used to:

[0041] If the first CPU architecture information is the same as the second CPU architecture information, the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized and calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0042] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0043] The memory is used to store computer programs;

[0044] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0045] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0046] The embodiments of the present invention include the following advantages:

[0047] In an embodiment of the present invention, it can be applied to a heterogeneous CPU hybrid deployment cluster, which includes several servers, each server includes several computing nodes. In the process of processing a data query task, the data query task can be decomposed in response to receiving the data query task to obtain an execution plan corresponding to the data query task, the execution plan includes several execution processes and a target operator corresponding to each execution process; each target operator is converted into a target vectorized operator, and each target vectorized operator is assigned to a corresponding target computing node, and the CPU architecture information corresponding to the target computing node is obtained to determine a target vectorized operator library that matches the CPU architecture information; it is determined whether the target vectorized operator involves reading migration data, and according to the determination result, the data processing process corresponding to the execution plan is executed through the target vectorized operator library to obtain the execution result corresponding to the execution plan, so that by converting the target operator involved in the execution plan into the corresponding vectorized operator, and performing data processing based on the vectorized operator, the computing performance of the heterogeneous environment cluster can be effectively improved and the efficiency of data processing can be ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of a method for processing data provided in an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of a calculation process provided in an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of a physical plan and Java code provided in an embodiment of the present invention;

[0051] Figure 4 is a schematic diagram of data processing provided in an embodiment of the present invention;

[0052] Figure 5 It is a structural block diagram of a data processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] As an example, vectorized computing technology significantly improves the efficiency of computing tasks by processing multiple data elements simultaneously. Among them, the SIMD (Single Instruction, Multiple Data) instruction set in the processor, such as AVX (Advanced Vector Extensions), SSE (Streaming SIMD Extensions) and NEON (Single Instruction, Multiple Data), can perform the same operation on multiple data elements in one instruction cycle. Vectorization technology is widely used in scientific computing, image processing, machine learning, and big data processing. For example, in machine learning, vectorized computing accelerates the matrix operations of neural networks; in big data processing frameworks such as Apache Spark, vectorization technology improves the efficiency of data reading and querying.

[0055] Among them, in the related technologies, Spark vectorized computing technologies, such as Gluten+Velox for X86 architecture and OmniOperator for ARM architecture, can usually run efficiently in a single architecture cluster. However, the existing computing frameworks of related technologies are limited to a single architecture (for example, Gluten can only be applied to X86 architecture, and OmniOperator can only be applied to ARM architecture). Since the computing process uses vectorized instruction sets and shuffle data storage formats that are strongly related to the underlying CPU architecture, the vectorized computing solutions of the current related technologies cannot be used in clusters with mixed deployment of heterogeneous CPUs.

[0056] In this regard, in the present invention, by dynamically selecting the vectorization library, shuffle data structure compatibility, and operator dictionary technology at runtime, the compatibility issues caused by different underlying CPU architectures can be shielded during the calculation process, solving the problem that the industry's Spark vectorization technology cannot be used in heterogeneous CPU clusters, and more efficiently using the overall computing resources of the cluster, and reducing the difficulty of differentiated management caused by the underlying CPU architecture. Specifically, in the process of processing the data query task, the data query task can be decomposed in response to receiving the data query task to obtain an execution plan corresponding to the data query task, the execution plan including several execution processes and a target operator corresponding to each execution process; each target operator is converted into a target vectorized operator, and each target vectorized operator is assigned to a corresponding target computing node, and the CPU architecture information corresponding to the target computing node is obtained to determine a target vectorized operator library that matches the CPU architecture information; it is determined whether the target vectorized operator involves reading migration data, and according to the determination result, the data processing process corresponding to the execution plan is executed through the target vectorized operator library to obtain the execution result corresponding to the execution plan, thereby converting the target operator involved in the execution plan into the corresponding vectorized operator and performing data processing based on the vectorized operator, which can effectively improve the computing performance of the heterogeneous environment cluster and ensure the efficiency of data processing.

[0057] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, some technical features involved in the embodiments of the present invention are explained below:

[0058] Heterogeneous architecture: Servers use different types of CPU processors, operating systems, etc.

[0059] Hybrid deployment: refers to the deployment of a server cluster using a heterogeneous architecture. Hybrid deployment can involve a combination of hardware, operating system, network, etc., aiming to make full use of various resources and technologies to meet specific needs.

[0060] Vectorized computing: Vectorized computing is a special type of parallel computing. Compared to general programs that only perform one operation at a time, it can perform multiple operations at the same time, usually executing the same instruction or a batch of instructions on different data, or applying instructions to an array / vector.

[0061] Spark: Apache Spark is an open source big data processing framework, mainly used for fast processing and analysis of large-scale data. It is based on in-memory computing and can significantly improve the efficiency of big data processing.

[0062] Gluten: The Gluten project can inject vectorized computing capabilities into the Spark engine, improving Spark's execution efficiency and cost.

[0063] Velox: A general-purpose vectorized computing engine based on the X86 architecture.

[0064] OmniOperator: A general-purpose vectorized computing engine based on the ARM architecture.

[0065] Shuffle: Shuffle refers to the process of moving data from one node to another during data processing. This process usually occurs when data needs to be regrouped or aggregated.

[0066] Vectorized engines: Vectorized engines in Spark SQL (such as Apache Arrow) improve computing efficiency by processing data in batches. Vectorized engines store data by column and process multiple data rows at a time, thereby reducing CPU instruction overhead.

[0067] Vectorized operators: Operators supported by the vectorized engine (such as vectorized filtering, vectorized aggregation, etc.) will utilize the CPU's SIMD (single instruction multiple data) instruction set during execution, thereby improving computing performance.

[0068] Vectorized operator library: Contains pre-optimized vectorized operator implementations that are optimized for different CPU architectures. The vectorized operator library selects the optimal operator implementation based on the CPU architecture of the current computing node.

[0069] Dynamic selection: In a heterogeneous CPU cluster, the vectorized operator library dynamically selects the appropriate operator implementation based on the CPU type of the computing node to fully utilize the characteristics of different CPU architectures.

[0070] It should be noted that for a heterogeneous CPU mixed deployment cluster, it can be composed of different types of heterogeneous servers, each server can include several computing nodes, and the computing node can be a logical unit on the server, responsible for executing specific computing tasks, such as a physical CPU core, a virtual CPU core or a container instance. Furthermore, the computing unit can be the smallest execution unit, which can be a CPU core or a thread, and the computing unit is responsible for executing specific operators. In addition, the operator can be the basic operation unit in Spark SQL, such as Filter, Aggregate, Join, etc. The operator defines the logic of data processing, and Spark SQL can allocate the operator to different computing nodes and computing units for execution according to the query plan. Happy selection, the scheduler will consider the heterogeneity of the CPU and allocate the operator suitable for the specific CPU architecture to the corresponding computing unit.

[0071] Reference Figure 1, shows a flowchart of the steps of a data processing method provided in an embodiment of the present invention, which is applied to a heterogeneous CPU hybrid deployment cluster, wherein the heterogeneous CPU hybrid deployment cluster includes a plurality of servers, each of which includes a plurality of computing nodes, and specifically may include the following steps:

[0072] Step 101, in response to receiving a data query task, decompose the data query task to obtain an execution plan corresponding to the data query task, wherein the execution plan includes a plurality of execution processes and a target operator corresponding to each of the execution processes;

[0073] In the embodiment of the present invention, the user can submit a corresponding data query task in the client, and the client can decompose the data query task to obtain a corresponding execution plan, wherein the execution plan may include several execution processes and a target operator corresponding to each execution process.

[0074] It should be noted that when processing data query tasks, the process from receiving the task to generating the execution plan involves multiple steps, including task decomposition, execution plan generation, operator selection, etc. The following is a detailed description of this process:

[0075] 1. Receive data query tasks

[0076] Task reception: The client receives a data query task from a user or an application. The query task may be a task submitted in the form of an SQL statement, such as SELECT, JOIN, GROUP BY, etc., which is not limited in the present invention.

[0077] 2. Task decomposition

[0078] Syntax parsing: First, the system parses the SQL statement and converts it into an abstract syntax tree. The parser can check whether the syntax of the SQL statement is correct and generate the corresponding abstract syntax tree.

[0079] Logical plan generation: Based on the generated abstract syntax tree, the client can generate the corresponding logical execution plan. The logical execution plan describes the logical operation of the query but does not involve specific execution details. The logical execution plan usually includes multiple logical operators, such as Filter, Project, Aggregate, etc.

[0080] 3. Execution plan generation

[0081] Furthermore, the client can optimize the logical plan. Specifically, after generating the logical execution plan, the client can optimize it. The optimizer configured in the client executes corresponding optimization strategies, such as predicate pushdown, column pruning, connection reordering, etc., to reduce the amount of data scanning and computational complexity. The optimized logical execution plan is converted into a physical execution plan, where the physical execution plan describes how to execute the query in a specific hardware and software environment. Optionally, the physical execution plan includes multiple physical operators, such as HashJoin, SortMergeJoin, BroadcastJoin, etc.

[0082] 4. Operator Selection

[0083] Target operator determination: In the physical execution plan, the client can select a suitable target operator for each execution process. The selection of the target operator depends on many factors, such as data distribution, hardware characteristics, optimization strategy, etc., which are not limited in the present invention.

[0084] After selecting the target operator, the client can further optimize the implementation of the target operator, such as selecting a vectorized operator, utilizing the SIMD instruction set, and performing memory optimization, to improve execution efficiency.

[0085] 5. Execution plan generation

[0086] Execution plan generation: The final generated execution plan includes several execution processes and the target operator corresponding to each execution process. The execution plan describes the complete execution path of the query task, including steps such as data reading, processing, transmission, and result output.

[0087] Through the above process, the data query task is decomposed into corresponding execution processes and corresponding target operators, so that the data query task can be distributedly processed through multiple computing nodes and computing units to improve the efficiency of data processing.

[0088] Step 102: convert each of the target operators into a target vectorization operator, and assign each of the target vectorization operators to a corresponding target computing node, and obtain CPU architecture information corresponding to the target computing node, and determine a target vectorization operator library that matches the CPU architecture information;

[0089] In an embodiment of the present invention, the target operator can be converted into a corresponding target vectorized operator so that the target vectorized operator can be assigned to the corresponding target computing node, and the target computing node can perform the calculation corresponding to the target vectorized operator. At the same time, before the calculation is formally performed, the CPU architecture information corresponding to the target computing node can be obtained to determine the target vectorized operator library that matches the CPU architecture information, thereby effectively reducing the network communication overhead and improving the computing efficiency by dynamically determining the vectorized operator library.

[0090] In some feasible implementations, before the target operator is vectorized, a vectorized operator dictionary for the target operator can be obtained. The vectorized operator dictionary maintains the vectorized operators supported by the computing nodes. If the target operator is in the vectorized operator dictionary and the CPU architectures in the computing nodes all support the target operator, the target operator is converted to the target vectorized operator; if the target operator is not in the vectorized operator dictionary and / or the CPU architecture in the computing node does not support the target operator, the native computing engine is used to execute the corresponding execution plan.

[0091] In the specific implementation, the vectorized operator dictionary maintains a list of vectorized operators supported by the computing node, which is used to determine whether the target operator can be vectorized. By checking whether the target operator exists in the vectorized operator dictionary, it is determined whether the target operator supports the vectorized computing feature. At the same time, it is checked whether all CPU architectures in the computing node support the vectorized computing of the target operator, ensuring that the target operator can be correctly executed on all computing nodes to avoid execution errors caused by incompatible CPU architectures. Specifically, if the target operator exists in the vectorized operator dictionary and all CPU architectures in the computing node support the target operator, the target operator is converted to a target vectorized operator to utilize the vectorized computing feature and improve computing performance; if the target operator does not exist in the vectorized operator dictionary, or some CPU architectures in the computing node do not support the target operator, the native computing engine is used to execute the corresponding execution plan to ensure that the computing task can be executed normally and avoid execution errors caused by the lack of support for vectorized computing. Then, through the corresponding vectorized feature judgment strategy, the execution of the vectorized computing task can be optimized to improve the overall computing efficiency and resource utilization.

[0092] In one example, before the target computing node formally performs vectorized computing, based on the determined CPU architecture information of the target computing node, if the CPU architecture information of the assigned target computing node is the x86 architecture, a first vectorized operator library corresponding to the x86 architecture is determined; if the CPU architecture information of the assigned target computing node is the ARM architecture, a second vectorized operator library corresponding to the ARM architecture is determined. Optionally, the first vectorized operator library may be a Velox library, and the second vectorized operator library may be an Omni library, which is not limited by the present invention.

[0093] Step 103, determining whether the target vectorization operator involves reading migration data, and executing the data processing process corresponding to the execution plan through the target vectorization operator library according to the determination result, to obtain the execution result corresponding to the execution plan.

[0094] In an embodiment of the present invention, during the process of vectorized calculation, it can be determined whether the target vectorized operator involves reading migration data, and based on the determination result, the data processing process corresponding to the execution plan is executed through the target vectorized operator library to obtain the execution result corresponding to the execution plan, thereby converting the target operator involved in the execution plan into the corresponding vectorized operator and performing data processing based on the vectorized operator, thereby effectively improving the computing performance of the heterogeneous environment cluster and ensuring the efficiency of data processing.

[0095] In some feasible implementations, the target computing node includes a Native engine, and the data processing method to be adopted can be determined by judging whether the target vectorization operator involves reading migration data. If the judgment result indicates that the target vectorization operator involves reading migration data, the first CPU architecture information corresponding to the upstream computing node in the target computing node and the second CPU architecture information corresponding to the downstream computing node are obtained, and data conversion is performed according to the first CPU architecture information and the second CPU architecture information. Based on the result of the data conversion, the data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain the execution result corresponding to the execution plan; if the judgment result indicates that the target vectorization operator does not involve reading migration data , then the target vectorized operator is unloaded to the Native engine of the target computing node, and the underlying vectorized instruction set is called through the vectorized operator library to execute the target vectorized operator to obtain the execution result corresponding to the execution plan, thereby detecting the CPU architecture of the upstream and downstream nodes to determine the compatibility of data between different CPU architectures for data conversion, and based on the data conversion, ensure that the data can be correctly transmitted and processed between different architectures. When the target vectorized operator does not involve reading migration data, the target vectorized operator can be directly unloaded from the distributed computing framework to the local Native engine for execution, utilizing local computing resources, reducing scheduling overhead and network communication overhead, and improving computing efficiency.

[0096] In a specific implementation, in a case involving data migration, if the first CPU architecture information is different from the second CPU architecture information, then while the downstream computing node pulls the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node, and the downstream vectorized data structure is vectorized through the vectorized operator library to obtain an execution result corresponding to the execution plan; if the first CPU architecture information is the same as the second CPU architecture information, the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized through the vectorized operator library to obtain an execution result corresponding to the execution plan.

[0097] For example, assuming that the upstream computing node is of x86 architecture and the downstream computing node is of ARM architecture, the CPU architectures of the two computing nodes are different, and the data structure needs to be converted before vectorized computing can be performed; assuming that both the upstream computing node and the downstream computing node are of x86 architecture or ARM architecture, there is no need to convert the data structure, and vectorized computing can be performed after pulling the corresponding data.

[0098] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention is not limited to this.

[0099] In an embodiment of the present invention, it can be applied to a heterogeneous CPU hybrid deployment cluster, which includes several servers, each server includes several computing nodes. In the process of processing a data query task, the data query task can be decomposed in response to receiving the data query task to obtain an execution plan corresponding to the data query task, the execution plan includes several execution processes and a target operator corresponding to each execution process; each target operator is converted into a target vectorized operator, and each target vectorized operator is assigned to a corresponding target computing node, and the CPU architecture information corresponding to the target computing node is obtained to determine a target vectorized operator library that matches the CPU architecture information; it is determined whether the target vectorized operator involves reading migration data, and according to the determination result, the data processing process corresponding to the execution plan is executed through the target vectorized operator library to obtain the execution result corresponding to the execution plan, so that by converting the target operator involved in the execution plan into the corresponding vectorized operator, and performing data processing based on the vectorized operator, the computing performance of the heterogeneous environment cluster can be effectively improved and the efficiency of data processing can be ensured.

[0100] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are used for exemplary description:

[0101] As an example, in an embodiment of the present invention, it is injected into the Spark task in the form of a plug-in. After the client submits the SQL task, the Driver side parses the task and generates an execution plan. Through the operator dictionary, the operator that can use the vectorization feature is converted into a vectorized operator, and then the execution plan is distributed to the executor side. Before executing the operator, the Task task thread on the executor side first determines the local CPU architecture. If it is an X86 architecture, the underlying operator library is dynamically set to use Velox; if the local machine is an ARM architecture, the underlying operator library is dynamically set to use OmniOperator. Through the strongly architecture-related operator library, the underlying vectorized instruction set is called to read and calculate the local data, and then returned to the native operator of Spark in the form of jni, and the native operator performs further operations.

[0102] In particular, if it is the task start stage, the operator library directly reads the original input data from the hard disk and converts it into a data structure unique to the operator for subsequent calculations; if it is the shuffle stage, when the downstream data pulls the upstream shuffle data, it will simultaneously determine the local CPU architecture and obtain the CPU architecture of the upstream node. If the architectures are the same, the data is directly pulled for subsequent processing. If the architectures are different, the data of the upstream node is converted into a data structure supported by the local vectorization library as needed during the pulling process, and subsequent calculations are performed.

[0103] For example, refer to Figure 2 , which shows a schematic diagram of a calculation process provided in an embodiment of the present invention, and the specific process may include:

[0104] 1.1: The client submits a SparkSQL task;

[0105] 1.2: SparkDriver parses SQL and generates an execution plan;

[0106] 1.3: Use the vectorized operator dictionary to determine whether the execution plan can use the vectorized computing feature;

[0107] 1.4: Determine whether the operator in the execution plan supports the vectorization feature. If all CPU architectures in the cluster support the operator, it is marked as supported, otherwise it is marked as not supported;

[0108] 1.5: If it is not supported, the Spark native computing engine is used directly to perform calculations and return results;

[0109] 1.6: If supported, convert the relevant operators in the execution plan into vectorized operators;

[0110] 1.7: Send the execution plan to the SparkExecutor executor (when the executor starts, it calls the operating system interface to obtain information such as the CPU architecture and saves it in the executor's local variables);

[0111] 1.8: The Task thread in the Executore executor starts executing the assigned operator;

[0112] 1.9: Before actually executing vectorized calculations, obtain CPU architecture information from the executor local variables and dynamically determine the vectorized operator library;

[0113] 1.10: If the CPU is X86 architecture, use the Velox library, if it is ARM architecture, use the Omni library;

[0114] 1.11: Determine whether the operator reads shuffle data. If not, directly unload the operator to the native engine, call the underlying vectorized instruction set, execute the vectorized operator, and return the result;

[0115] 1.12: If it is shuffle data, determine whether the CPU architecture of the upstream and downstream nodes is the same. If they are the same, the downstream node directly pulls the shuffle intermediate data of the upstream node, performs vectorized calculations, and returns the results;

[0116] 1.13: If the CPU architectures of the upstream and downstream nodes are different, the downstream node will convert the data structure while pulling the shuffle data from the upstream node, converting the upstream vectorized data structure into the downstream vectorized data structure. After the conversion is completed, vectorized calculations are performed and the results are returned.

[0117] For example, suppose the user queries the following SQL statement in the client: SELECT sex, count(1) FROM STUDENT WHERE age>14 GROUP BY sex. Find all students older than 14 from the STUDENT table, group these students by gender, and calculate the number of male and female students respectively. The vectorized calculation process provided in the above embodiment can perform lexical analysis and grammatical analysis on the SQL to generate a physical plan tree. The logical representation of the physical plan tree and the corresponding Java code are as follows: Figure 3 Specifically, the Java code corresponding to the VectortableScan operator reads the table file line by line, stores it in the collection iterator, and uses it downstream; the VectorFilter operator obtains the upstream iterator, scans the data line by line, filters the data with age greater than 14, and groups the data by sex locally, and saves it in a local temporary shuffle file; the task of the VectorGroup operator pulls the shuffle data from the server where the upstream operator task is located, performs aggregation calculations in the local memory, and finally displays the output.

[0118] Among them, refer to Figure 4 , which shows a schematic diagram of data processing provided in an embodiment of the present invention. In the process of executing vectorized calculations, thread Task1 (x86), thread Task2 (ARM), and thread Task3 (ARM) can read corresponding data from the data table STUDENT, respectively, such as:

[0119] Data read by thread Task1(x86):

[0120] Zhang Yi, 15, male

[0121] Li Yi, 16, male

[0122] Wang Yi, 13, male

[0123] Yan Zi, 15, female

[0124] Data read by thread Task2 (ARM):

[0125] Zhang Er, 16, male

[0126] Li Er, 13, male

[0127] Wang Er, 13, male

[0128] Parrot, 16, female

[0129] Data read by thread Task3 (ARM):

[0130] Zhang San, 13, male

[0131] Li San, 15, male

[0132] Wang San, 15, male

[0133] Eagle, 13, female

[0134] Next, after reading the corresponding data from the data table, each thread can perform the corresponding vector calculation to extract the data with an age greater than 14. The extracted data can be:

[0135] The data extracted by thread Task1(x86):

[0136] Zhang Yi, 15, male

[0137] Li Yi, 16, male

[0138] Yan Zi, 15, female

[0139] Data extracted by thread Task2 (ARM):

[0140] Zhang Er, 16, male

[0141] Parrot, 16, female

[0142] Data extracted by thread Task3 (ARM):

[0143] Li San, 15, male

[0144] Wang San, 15, male

[0145] At the same time, when data is migrated between different computing nodes, it can be determined whether the data structure needs to be converted based on whether the CPU architecture is the same. For example, Task1 (x86) and Task1 (ARM) belong to different CPU architectures. In the process of pulling, x86ShuffleData needs to be converted to armShuffleData. Conversely, the data structure conversion is also performed. When the CPU architecture is the same, data can be pulled directly, and after the data is pulled, the corresponding vectorized calculation is performed and the corresponding calculation results are returned. For example, the final output result can be: male, 5; female 2, that is, there are 5 students older than 14 and 2 students older than 14.

[0146] In the above process, the local architecture information is used before the vectorized operator is actually unloaded to the native library for calculation, so that the vectorized library to be used can be dynamically determined; and through the transformation of the Spark shuffle logic, the compatibility of the intermediate data structures between operator libraries of different architectures is ensured during the transmission process; and the global vectorized operator dictionary is maintained to record the vectorized operators that can be supported by all CPU architectures in the cluster; and the computing performance of heterogeneous environment clusters is improved through vectorized computing technology, so there is no need to pay attention to the underlying CPU architecture information. The vectorized computing characteristics of Spark are used to improve computing efficiency while reducing the difficulty of resource management of heterogeneous CPU clusters.

[0147] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0148] Reference Figure 5 , shows a structural block diagram of a data processing device provided in an embodiment of the present invention, which is applied to a heterogeneous CPU hybrid deployment cluster, wherein the heterogeneous CPU hybrid deployment cluster includes a plurality of servers, each of which includes a plurality of computing nodes, and specifically may include the following modules:

[0149] The task decomposition module 501 is used to, in response to receiving a data query task, decompose the data query task to obtain an execution plan corresponding to the data query task, wherein the execution plan includes a plurality of execution processes and a target operator corresponding to each execution process;

[0150] Configuration module 502, used to convert each of the target operators into a target vectorization operator, and assign each of the target vectorization operators to a corresponding target computing node, and obtain CPU architecture information corresponding to the target computing node, and determine a target vectorization operator library matching the CPU architecture information;

[0151] The execution module 503 is used to determine whether the target vectorization operator involves reading migration data, and execute the data processing process corresponding to the execution plan through the target vectorization operator library according to the determination result to obtain the execution result corresponding to the execution plan.

[0152] In some feasible implementations, the configuration module 502 is specifically used to:

[0153] Obtaining a vectorized operator dictionary for the target operator, wherein the vectorized operator dictionary maintains vectorized operators supported by the computing node;

[0154] If the target operator is located in the vectorized operator dictionary and the CPU architectures in the computing nodes all support the target operator, the target operator is converted into a target vectorized operator.

[0155] Some possible implementations also include:

[0156] A computing execution module is used to use a native computing engine to execute the execution plan if the target operator is not located in the vectorized operator dictionary and / or the CPU architecture in the computing node does not support the target operator.

[0157] In some feasible implementations, the configuration module 502 is specifically used to:

[0158] If the CPU architecture information is an x86 architecture, determining a first vectorization operator library corresponding to the x86 architecture;

[0159] If the CPU architecture information is an ARM architecture, a second vector operator library corresponding to the ARM architecture is determined.

[0160] In some feasible implementations, the target computing node includes a Native engine, and the execution module 503 is specifically used to:

[0161] If the judgment result indicates that the target vectorization operator involves reading migration data, first CPU architecture information corresponding to an upstream computing node in the target computing node and second CPU architecture information corresponding to a downstream computing node are obtained, data conversion is performed according to the first CPU architecture information and the second CPU architecture information, and based on the result after the data conversion, a data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain an execution result corresponding to the execution plan;

[0162] If the judgment result indicates that the target vectorization operator does not involve reading migration data, the target vectorization operator is unloaded to the Native engine of the target computing node, and the underlying vectorization instruction set is called through the vectorization operator library to execute the target vectorization operator to obtain the execution result corresponding to the execution plan.

[0163] In some feasible implementations, the execution module 503 is specifically used to:

[0164] If the first CPU architecture information is different from the second CPU architecture information, when the downstream computing node pulls the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node, and the downstream vectorized data structure is vectorized by the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0165] In some feasible implementations, the execution module 503 is specifically used to:

[0166] If the first CPU architecture information is the same as the second CPU architecture information, the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized and calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0167] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0168] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0169] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned data processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0170] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0171] It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program codes.

[0172] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0175] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0176] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0177] The data processing method and the data processing device provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and the scope of application. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data processing method, characterized in that: Applied to a heterogeneous CPU mixed deployment cluster, the heterogeneous CPU mixed deployment cluster includes a plurality of servers, each of the servers includes a plurality of computing nodes, the method includes: In response to receiving a data query task, decomposing the data query task to obtain an execution plan corresponding to the data query task, wherein the execution plan includes a plurality of execution processes and a target operator corresponding to each of the execution processes; Convert each of the target operators into a target vectorization operator, and assign each of the target vectorization operators to a corresponding target computing node, and obtain CPU architecture information corresponding to the target computing node, and determine a target vectorization operator library that matches the CPU architecture information; It is determined whether the target vectorization operator involves reading migration data, and according to the determination result, a data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain an execution result corresponding to the execution plan.

2. The method according to claim 1, characterized in that The converting each of the target operators into a target vectorization operator comprises: Obtaining a vectorized operator dictionary for the target operator, wherein the vectorized operator dictionary maintains vectorized operators supported by the computing node; If the target operator is located in the vectorized operator dictionary and the CPU architectures in the computing nodes all support the target operator, the target operator is converted into a target vectorized operator.

3. The method according to claim 2, characterized in that Also includes: If the target operator is not located in the vectorized operator dictionary, and / or the CPU architecture in the computing node does not support the target operator, a native computing engine is used to execute the execution plan.

4. The method according to claim 1, characterized in that: The determining of a target vectorization operator library matching the CPU architecture information includes: If the CPU architecture information is an x86 architecture, determining a first vectorization operator library corresponding to the x86 architecture; If the CPU architecture information is an ARM architecture, a second vector operator library corresponding to the ARM architecture is determined.

5. The method according to claim 1, characterized in that The target computing node includes a Native engine, and executing a data processing process corresponding to the execution plan through the target vectorized operator library according to the judgment result to obtain an execution result corresponding to the execution plan includes: If the judgment result indicates that the target vectorization operator involves reading migration data, first CPU architecture information corresponding to an upstream computing node in the target computing node and second CPU architecture information corresponding to a downstream computing node are obtained, data conversion is performed according to the first CPU architecture information and the second CPU architecture information, and based on the result after the data conversion, a data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain an execution result corresponding to the execution plan; If the judgment result indicates that the target vectorization operator does not involve reading migration data, the target vectorization operator is unloaded to the Native engine of the target computing node, and the underlying vectorization instruction set is called through the vectorization operator library to execute the target vectorization operator to obtain the execution result corresponding to the execution plan.

6. The method according to claim 5, characterized in that The performing data conversion according to the first CPU architecture information and the second CPU architecture information, and executing the data processing process corresponding to the execution plan through the target vectorized operator library based on the result of the data conversion, to obtain the execution result corresponding to the execution plan, includes: If the first CPU architecture information is different from the second CPU architecture information, when the downstream computing node pulls the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node, and the downstream vectorized data structure is vectorized calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

7. The method according to claim 6, characterized in that: The performing data conversion according to the first CPU architecture information and the second CPU architecture information, and executing the data processing process corresponding to the execution plan through the target vectorized operator library based on the result of the data conversion, to obtain the execution result corresponding to the execution plan, includes: If the first CPU architecture information is the same as the second CPU architecture information, the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized and calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

8. A data processing device, characterized in that: Applied to a heterogeneous CPU mixed deployment cluster, the heterogeneous CPU mixed deployment cluster includes a plurality of servers, each of the servers includes a plurality of computing nodes, and the device includes: A task decomposition module, configured to, in response to receiving a data query task, decompose the data query task to obtain an execution plan corresponding to the data query task, wherein the execution plan includes a plurality of execution processes and a target operator corresponding to each of the execution processes; A configuration module, used to convert each of the target operators into a target vectorization operator, and assign each of the target vectorization operators to a corresponding target computing node, and obtain CPU architecture information corresponding to the target computing node, and determine a target vectorization operator library that matches the CPU architecture information; An execution module is used to determine whether the target vectorization operator involves reading migration data, and execute the data processing process corresponding to the execution plan through the target vectorization operator library according to the determination result to obtain the execution result corresponding to the execution plan.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images