Data processing method and apparatus, electronic device and storage medium

Abstract

Provided in the embodiments of the present application are a data processing method and apparatus, an electronic device and a storage medium. The method comprises: in response to receiving a data query task, decomposing the data query task to obtain an execution plan corresponding to the data query task, the execution plan comprising a plurality of execution procedures and a target operator that corresponds to each execution procedure; converting each target operator into a target vectorized operator, allocating each target vectorized operator to a corresponding target computing node, and acquiring CPU architecture information corresponding to the target computing nodes, and determining a target vectorized operator library matching the CPU architecture information; and determining whether the target vectorized operators involve reading migrated data, and on the basis of the determination result and by means of the target vectorized operator library, executing the data processing procedures corresponding to the execution plan, so as to obtain an execution result corresponding to the execution plan.

Description

Data processing methods, devices, electronic equipment and storage media

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on December 4, 2024, application number 202411775051.1, entitled "Data processing method, apparatus, electronic device and storage medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of big data technology, and in particular to a data processing method, a data processing device, an electronic device, and a computer-readable storage medium. Background Technology

[0004] With the development of big data technology, vectorized computing technology has significantly improved the efficiency of computing tasks by processing multiple data elements simultaneously. Vectorization technology is widely used in scientific computing, image processing, machine learning, and big data processing. For example, in machine learning, vectorized computing accelerates matrix operations in neural networks; in big data processing frameworks such as Apache Spark, vectorization technology improves the efficiency of data reading and querying. However, in clusters with heterogeneous CPUs, vectorization technology cannot be used for corresponding data processing, greatly limiting its application and hindering its effectiveness in improving data processing efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium to solve or partially solve the problem that vectorized computing has application limitations in heterogeneous CPU hybrid deployment clusters, resulting in the inability to effectively improve data processing efficiency.

[0006] This application discloses a data processing method applied to a heterogeneous CPU hybrid deployment cluster, wherein the heterogeneous CPU hybrid deployment cluster includes several servers, and each server includes several computing nodes. The method includes:

[0007] In response to receiving a data query task, the data query task is decomposed to obtain an execution plan corresponding to the data query task. The execution plan includes several execution processes and a target operator corresponding to each execution process.

[0008] Each of the target operators is converted into target vectorized operators, and each of the target vectorized operators is assigned to the corresponding target computing nodes. The CPU architecture information corresponding to the target computing nodes is obtained, and a target vectorized operator library matching the CPU architecture information is determined.

[0009] Determine whether the target vectorization operator involves reading migration data, and based on the determination result, execute the data processing process corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan.

[0010] In some feasible implementations, the conversion of each of the target operators into target vectorized operators includes:

[0011] Obtain a vectorized operator dictionary for the target operator, wherein the vectorized operator dictionary maintains the vectorized operators supported by the computing node;

[0012] If the target operator is located in the vectorized operator dictionary, and the CPU architecture in the computing node supports the target operator, then the target operator is converted into a target vectorized operator.

[0013] Among some feasible implementation methods are:

[0014] If the target operator is not in the vectorized operator dictionary, and / or the CPU architecture in the computing node does not support the target operator, then the execution plan is executed using the native computing engine.

[0015] In some feasible implementations, determining the target vectorized operator library that matches the CPU architecture information includes:

[0016] If the CPU architecture information is x86 architecture, then determine the first vectorized operator library corresponding to the x86 architecture;

[0017] If the CPU architecture information is ARM architecture, then determine the second vector operator library corresponding to the ARM architecture.

[0018] In some feasible implementations, the target computing node includes a native engine. The step of executing the data processing procedure corresponding to the execution plan through the target vectorized operator library based on the judgment result to obtain the execution result corresponding to the execution plan includes:

[0019] If the judgment result indicates that the target vectorization operator involves reading migration data, then the first CPU architecture information corresponding to the upstream computing node and the second CPU architecture information corresponding to the downstream computing node in the target computing node are obtained. Data conversion is performed according to the first CPU architecture information and the second CPU architecture information. Based on the result of the data conversion, the data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain the execution result corresponding to the execution plan.

[0020] If the judgment result indicates that the target vectorization operator does not involve reading migration data, then the target vectorization operator is unloaded into the native engine of the target computing node, and the underlying vectorization instruction set is called through the vectorization operator library to execute the target vectorization operator and obtain the execution result corresponding to the execution plan.

[0021] In some feasible implementations, the step of performing data conversion based on the first CPU architecture information and the second CPU architecture information, and then, based on the converted data, executing a data processing procedure corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan, includes:

[0022] If the first CPU architecture information is different from the second CPU architecture information, then while the downstream computing node is pulling the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node. The vectorized operator library is used to perform vectorized calculations on the downstream vectorized data structure to obtain the execution result corresponding to the execution plan.

[0023] In some feasible implementations, the step of performing data conversion based on the first CPU architecture information and the second CPU architecture information, and then, based on the converted data, executing a data processing procedure corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan, includes:

[0024] If the first CPU architecture information is the same as the second CPU architecture information, then the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized and calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0025] This application also discloses a data processing apparatus applied to a heterogeneous CPU hybrid deployment cluster, wherein the heterogeneous CPU hybrid deployment cluster includes a plurality of servers, each server including a plurality of computing nodes, and the apparatus includes:

[0026] The task decomposition module is used to decompose the data query task in response to receiving the data query task, and obtain the execution plan corresponding to the data query task. The execution plan includes several execution processes and a target operator corresponding to each execution process.

[0027] The configuration module is used to convert each of the target operators into target vectorized operators, allocate each of the target vectorized operators to the corresponding target computing nodes, and obtain the CPU architecture information corresponding to the target computing nodes to determine the target vectorized operator library that matches the CPU architecture information.

[0028] The execution module is used to determine whether the target vectorization operator involves reading migration data, and based on the determination result, to execute the data processing process corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan.

[0029] In some feasible implementations, the configuration module is specifically used for:

[0030] Obtain a vectorized operator dictionary for the target operator, wherein the vectorized operator dictionary maintains the vectorized operators supported by the computing node;

[0031] If the target operator is located in the vectorized operator dictionary, and the CPU architecture in the computing node supports the target operator, then the target operator is converted into a target vectorized operator.

[0032] Among some feasible implementation methods are:

[0033] The computation execution module is configured to execute the execution plan using the native computation engine if the target operator is not located in the vectorized operator dictionary and / or the CPU architecture in the computation node does not support the target operator.

[0034] In some feasible implementations, the configuration module is specifically used for:

[0035] If the CPU architecture information is x86 architecture, then determine the first vectorized operator library corresponding to the x86 architecture;

[0036] If the CPU architecture information is ARM architecture, then determine the second vector operator library corresponding to the ARM architecture.

[0037] In some feasible implementations, the target computing node includes a native engine, and the execution module is specifically used for:

[0038] If the judgment result indicates that the target vectorization operator involves reading migration data, then the first CPU architecture information corresponding to the upstream computing node and the second CPU architecture information corresponding to the downstream computing node in the target computing node are obtained. Data conversion is performed according to the first CPU architecture information and the second CPU architecture information. Based on the result of the data conversion, the data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain the execution result corresponding to the execution plan.

[0039] If the judgment result indicates that the target vectorization operator does not involve reading migration data, then the target vectorization operator is unloaded into the native engine of the target computing node, and the underlying vectorization instruction set is called through the vectorization operator library to execute the target vectorization operator and obtain the execution result corresponding to the execution plan.

[0040] In some feasible implementations, the execution module is specifically used for:

[0041] If the first CPU architecture information is different from the second CPU architecture information, then while the downstream computing node is pulling the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node. The vectorized operator library is used to perform vectorized calculations on the downstream vectorized data structure to obtain the execution result corresponding to the execution plan.

[0042] In some feasible implementations, the execution module is specifically used for:

[0043] If the first CPU architecture information is the same as the second CPU architecture information, then the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized and calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0044] This application also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0045] The memory is used to store computer programs;

[0046] When the processor executes a program stored in the memory, it implements the method described in the embodiments of this application.

[0047] This application also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this application.

[0048] The embodiments of the present invention have the following advantages:

[0049] In this embodiment, it can be applied to heterogeneous CPU hybrid deployment clusters. The heterogeneous CPU hybrid deployment cluster includes several servers, each server including several computing nodes. During the processing of data query tasks, in response to receiving a data query task, the data query task is decomposed to obtain an execution plan corresponding to the data query task. The execution plan includes several execution processes and target operators corresponding to each execution process. Each target operator is converted into a target vectorized operator, and each target vectorized operator is allocated to the corresponding target computing node. The CPU architecture information corresponding to the target computing node is obtained, and a target vectorized operator library matching the CPU architecture information is determined. It is determined whether the target vectorized operator involves reading migration data, and based on the determination result, the data processing process corresponding to the execution plan is executed through the target vectorized operator library to obtain the execution result corresponding to the execution plan. Therefore, by converting the target operators involved in the execution plan into corresponding vectorized operators and performing data processing based on the vectorized operators, the computing performance of the heterogeneous environment cluster can be effectively improved, ensuring the efficiency of data processing.

[0050] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.

[0052] Figure 1 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0053] Figure 2 is a schematic diagram of the calculation process provided in the embodiments of this application;

[0054] Figure 3 is a schematic diagram of the physical plan and Java code provided in the embodiments of this application;

[0055] Figure 4 is a schematic diagram of the data processing provided in the embodiments of this application;

[0056] Figure 5 is a structural block diagram of a data processing device provided in an embodiment of this application. Detailed Implementation

[0057] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] As an example, vectorized computing techniques significantly improve the efficiency of computational tasks by processing multiple data elements simultaneously. Specifically, SIMD (Single Instruction, Multiple Data) instruction sets in processors, such as AVX (Advanced Vector Extensions), SSE (Streaming SIMD Extensions), and NEON (Single Instruction, Multiple Data), can perform the same operation on multiple data elements within a single instruction cycle. Vectorization techniques are widely used in scientific computing, image processing, machine learning, and big data processing. For instance, in machine learning, vectorized computing accelerates matrix operations in neural networks; in big data processing frameworks such as Apache Spark, vectorization techniques improve the efficiency of data reading and querying.

[0059] Among the related technologies, Spark's vectorized computing techniques, such as Gluten+Velox for x86 architecture and OmniOperator for ARM architecture, can typically run efficiently in single-architecture clusters. However, the computing frameworks in these technologies are limited to a single architecture (e.g., Gluten can only be applied to x86 architecture, and OmniOperator can only be applied to ARM architecture). Because the computation process uses vectorized instruction sets and shuffle data storage formats that are strongly dependent on the underlying CPU architecture, the current vectorized computing solutions in these technologies cannot be used in clusters with heterogeneous CPUs.

[0060] In this application, by dynamically selecting the vectorization library at runtime, ensuring shuffle data structure compatibility, and employing operator dictionary technology, the incompatibility issues caused by different underlying CPU architectures can be shielded during computation. This solves the problem that Spark vectorization technology cannot be used in heterogeneous CPU clusters, thereby enabling more efficient use of the overall cluster computing resources and reducing the difficulty of managing the differences caused by the underlying CPU architecture. In some embodiments of this application, during the processing of data processing tasks (e.g., data query tasks), the data processing task can be decomposed in response to the receipt of the data processing task to obtain the execution plan corresponding to the data processing task. The execution plan includes several execution processes and target operators corresponding to each execution process. Each target operator is converted into a target vectorized operator, and each target vectorized operator is allocated to the corresponding target computing node. The CPU architecture information corresponding to the target computing node is obtained, and a target vectorized operator library matching the CPU architecture information is determined. It is determined whether the target vectorized operator involves reading migration data, and the data processing process corresponding to the execution plan is executed through the target vectorized operator library according to the determination result, and the execution result corresponding to the execution plan is obtained. Thus, by converting the target operators involved in the execution plan into corresponding vectorized operators and performing data processing based on the vectorized operators, the computing performance of the heterogeneous environment cluster can be effectively improved, and the efficiency of data processing can be guaranteed.

[0061] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, some technical features involved in the embodiments of this application are explained below:

[0062] Heterogeneous architecture: refers to servers using different types of CPU processors, operating systems, etc.

[0063] Hybrid deployment refers to deploying a cluster of servers with heterogeneous architectures. Hybrid deployment can involve combinations of hardware, operating systems, networks, etc., and aims to make full use of various resources and technologies to meet specific needs.

[0064] Vectorized computation: Vectorized computation is a special kind of parallel computation. Compared with the way a general program executes only one operation at a time, it can execute multiple operations at the same time. It usually executes the same one or a batch of instructions on different data, or applies instructions to an array / vector.

[0065] Spark: Apache Spark is an open-source big data processing framework primarily used for the rapid processing and analysis of large-scale data. It is based on in-memory computing, which can significantly improve the efficiency of big data processing.

[0066] Gluten: The Gluten project provides the Spark engine with vectorized computation capabilities, improving Spark's execution efficiency and cost.

[0067] Velox: A general-purpose vectorized computing engine based on the x86 architecture.

[0068] OmniOperator: A general-purpose vectorized computing engine based on the ARM architecture.

[0069] Shuffle: Shuffle refers to the process of moving data from one node to another during data processing. This process typically occurs when data needs to be regrouped or aggregated.

[0070] Vectorized engines: Vectorized engines in Spark SQL (such as Apache Arrow) improve computational efficiency by batch processing data. Vectorized engines store data column-wise and process multiple rows at a time, thus reducing CPU instruction overhead.

[0071] Vectorized operators: The operators supported by the vectorization engine (such as vectorized filtering, vectorized aggregation, etc.) will utilize the CPU's SIMD (Single Instruction Multiple Data) instruction set during execution, thereby improving computational performance.

[0072] Vectorized Operator Library: Contains pre-optimized vectorized operator implementations, which are optimized for different CPU architectures. The vectorized operator library selects the optimal operator implementation based on the CPU architecture of the current computing node.

[0073] Dynamic selection: In heterogeneous CPU clusters, the vectorized operator library dynamically selects the appropriate operator implementation based on the CPU type of the computing node to fully utilize the characteristics of different CPU architectures.

[0074] It's important to note that a heterogeneous CPU hybrid deployment cluster can consist of heterogeneous servers of different types. Each server can include several compute nodes, which can be logical units on the server responsible for executing specific computational tasks, such as physical CPU cores, virtual CPU cores, or container instances. Furthermore, a compute unit can be the smallest execution unit, such as a CPU core or a thread, responsible for executing specific operators. In addition, operators can be basic operation units in Spark SQL, such as Filter, Aggregate, and Join. Operators define the logic of data processing, and Spark SQL can allocate operators to different compute nodes and compute units for execution based on the query plan. In some implementations, the scheduler considers CPU heterogeneity and allocates operators suitable for a specific CPU architecture to the corresponding compute units.

[0075] Referring to Figure 1, a flowchart illustrating a data processing method provided in an embodiment of this application is shown. This method is applied to a heterogeneous CPU hybrid deployment cluster, which includes several servers, each server including several computing nodes. The method may include the following steps 101-103.

[0076] Step 101: In response to receiving a data processing task, the data processing task is decomposed to obtain an execution plan corresponding to the data processing task. The execution plan includes several execution processes and a target operator corresponding to each execution process.

[0077] In some embodiments, the data processing task may be a data query task. This application uses a data query task as an example to illustrate the technical solution, but it is not intended to be limiting.

[0078] In other embodiments, the data processing task may also be a data modification task, etc., and this application does not impose specific limitations.

[0079] For example, in some embodiments of this application, a user can submit a corresponding data query task in the client, and the client can decompose the data query task to obtain a corresponding execution plan. The execution plan may include several execution processes and a target operator corresponding to each execution process.

[0080] It should be noted that processing data query tasks involves multiple steps from receiving the task to generating an execution plan, including task decomposition, execution plan generation, and operator selection. The following is a detailed description of this process:

[0081] 1. Receive data query tasks

[0082] Task Reception: The client receives a data query task from a user or application. The query task can be submitted in the form of SQL statements, such as SELECT, JOIN, GROUP BY, etc., and this application does not impose any restrictions on this.

[0083] 2. Task Breakdown

[0084] Syntax parsing: First, the SQL statement is parsed, converting it into an abstract syntax tree. The parser checks the syntax of the SQL statement for correctness and generates the corresponding abstract syntax tree.

[0085] Logical plan generation: Based on the generated abstract syntax tree, the client can generate a corresponding logical execution plan. The logical execution plan describes the logical operations of the query, but does not involve the specific execution details. A logical execution plan typically includes multiple logical operators, such as Filter, Project, and Aggregate.

[0086] 3. Execution plan generation

[0087] Furthermore, the client can optimize the logical execution plan. Specifically, after generating the logical execution plan, the client can optimize it. The optimizer configured in the client executes corresponding optimization strategies, such as predicate pushdown, column pruning, and join reordering, to reduce data scanning volume and computational complexity. The optimized logical execution plan is then converted into a physical execution plan, which describes how the query will be executed in the specific hardware and software environment. Optionally, the physical execution plan includes multiple physical operators, such as HashJoin, SortMergeJoin, and BroadcastJoin.

[0088] 4. Operator Selection

[0089] Target operator determination: In the physical execution plan, the client can select a suitable target operator for each execution process. The selection of the target operator depends on various factors, such as data distribution, hardware characteristics, and optimization strategies, which are not limited in this application. The target operator refers to the physical implementation algorithm selected by the client for each operation step in the logical execution plan.

[0090] After selecting the target operator, the client can further optimize the implementation of the target operator, such as by selecting a vectorized operator, utilizing the SIMD instruction set, and performing memory optimization, in order to improve execution efficiency.

[0091] 5. Execution Plan Generation

[0092] Execution plan generation: The final generated execution plan includes several execution processes and the target operator corresponding to each process. The execution plan describes the complete execution path of the query task, including steps such as data reading, processing, transmission, and result output.

[0093] The above process decomposes the data query task into corresponding execution processes and target operators, so that the data query task can be processed in a distributed manner through multiple computing nodes and computing units, thereby improving the efficiency of data processing.

[0094] Step 102: Convert each of the target operators into target vectorized operators, allocate each of the target vectorized operators to the corresponding target computing nodes, and obtain the CPU architecture information corresponding to the target computing nodes to determine the target vectorized operator library that matches the CPU architecture information;

[0095] In some embodiments of this application, the target operator can be converted into a corresponding target vectorized operator so that the target vectorized operator can be allocated to the corresponding target computing node, and the target computing node can perform the computation corresponding to the target vectorized operator. Furthermore, before the actual computation, the target vectorized operator library matching the CPU architecture information can be determined by obtaining the CPU architecture information of the target computing node. This dynamic determination of the vectorized operator library effectively reduces network communication overhead and improves computational efficiency.

[0096] In some feasible implementations, before vectorizing the target operator, a dictionary of vectorized operators for the target operator can be obtained. The dictionary of vectorized operators maintains the vectorized operators supported by the computing nodes. If the target operator is in the dictionary of vectorized operators and the CPU architecture of the computing nodes all supports the target operator, then the target operator is converted into a target vectorized operator. If the target operator is not in the dictionary of vectorized operators and / or the CPU architecture of the computing nodes does not support the target operator, then the native computing engine is used to execute the corresponding execution plan.

[0097] In the specific implementation, the vectorized operator dictionary maintains a list of vectorized operators supported by computing nodes. This list is used to determine whether a target operator can be vectorized. By checking if the target operator exists in the dictionary, it is determined whether the target operator supports vectorized computation features. Simultaneously, it is checked whether all CPU architectures in the computing node support vectorized computation of the target operator, ensuring that the target operator can be correctly executed on all computing nodes and avoiding execution errors caused by CPU architecture incompatibility. Specifically, if the target operator exists in the dictionary and all CPU architectures in the computing node support it, the target operator is converted into a target vectorized operator to leverage vectorized computation features and improve computational performance. If the target operator does not exist in the dictionary, or some CPU architectures in the computing node do not support it, the native computing engine is used to execute the corresponding execution plan to ensure that the computation task can be executed normally, avoiding execution errors caused by lack of vectorized computation support. Thus, through appropriate vectorization feature judgment strategies, the execution of vectorized computation tasks can be optimized, improving overall computational efficiency and resource utilization.

[0098] In one example, before the target computing node formally performs vectorized computation, based on the determined CPU architecture information of the target computing node, if the allocated target computing node's CPU architecture information is x86 architecture, then a first vectorization operator library corresponding to the x86 architecture is determined; if the allocated target computing node's CPU architecture information is ARM architecture, then a second vectorization operator library corresponding to the ARM architecture is determined. Optionally, the first vectorization operator library can be the Velox library, and the second vectorization operator library can be the Omni library; this application does not impose any restrictions on this.

[0099] Step 103: Determine whether the target vectorization operator involves reading migration data, and based on the determination result, execute the data processing process corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan.

[0100] In this embodiment of the application, during the vectorized computation process, it can be determined whether the target vectorized operator involves reading migration data, and based on the determination result, the data processing process corresponding to the execution plan is executed through the target vectorized operator library to obtain the execution result corresponding to the execution plan. Thus, by converting the target operator involved in the execution plan into the corresponding vectorized operator and performing data processing based on the vectorized operator, the computing performance of the heterogeneous environment cluster can be effectively improved and the efficiency of data processing can be guaranteed.

[0101] In some feasible implementations, the target compute node contains a native engine. The data processing method can be determined by whether the target vectorization operator involves reading migration data. If the determination indicates that the target vectorization operator involves reading migration data, the first CPU architecture information corresponding to the upstream compute node and the second CPU architecture information corresponding to the downstream compute node in the target compute node are obtained. Data transformation is performed based on the first and second CPU architecture information, and based on the transformed data, the data processing procedure corresponding to the execution plan is executed through the target vectorization operator library to obtain the execution result corresponding to the execution plan. If the determination indicates that the target vectorization operator does not involve reading migration data... The target vectorized operator is then unloaded into the native engine of the target computing node. The underlying vectorized instruction set is called through the vectorized operator library to execute the target vectorized operator and obtain the execution result corresponding to the execution plan. By detecting the CPU architecture of upstream and downstream nodes, the compatibility of data between different CPU architectures can be determined so as to perform data conversion. Based on the data conversion, it is ensured that data can be correctly transmitted and processed between different architectures. When the target vectorized operator does not involve reading migration data, the target vectorized operator can be directly unloaded from the distributed computing framework to the local native engine for execution, utilizing local computing resources, reducing scheduling overhead and network communication overhead, and improving computing efficiency.

[0102] In specific implementations, when data migration is involved, if the first CPU architecture information differs from the second CPU architecture information, then while the downstream computing node pulls the migration data from the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node. The vectorized data structure is then vectorized using a vectorized operator library to obtain the execution result corresponding to the execution plan. If the first CPU architecture information is the same as the second CPU architecture information, then the downstream computing node pulls the target data from the upstream computing node, and the target data is vectorized using a vectorized operator library to obtain the execution result corresponding to the execution plan.

[0103] For example, if the upstream computing node is an x86 architecture and the downstream computing node is an ARM architecture, the CPU architectures of the two computing nodes are different, and data structure conversion is required before vectorized computation can be performed. If both the upstream and downstream computing nodes are x86 or ARM architectures, no data structure conversion is required, and vectorized computation can be performed after the corresponding data is retrieved.

[0104] It should be noted that the embodiments of this application include, but are not limited to, the examples described above. It is understood that those skilled in the art can make further settings according to actual needs under the guidance of the ideas in the embodiments of this application, and this application does not impose any restrictions on this.

[0105] In this embodiment, it can be applied to heterogeneous CPU hybrid deployment clusters. The heterogeneous CPU hybrid deployment cluster includes several servers, each server including several computing nodes. During the processing of data processing tasks, in response to receiving a data processing task, the data processing task is decomposed to obtain an execution plan corresponding to the data processing task. The execution plan includes several execution processes and target operators corresponding to each execution process. Each target operator is converted into a target vectorized operator, and each target vectorized operator is allocated to the corresponding target computing node. The CPU architecture information corresponding to the target computing node is obtained, and a target vectorized operator library matching the CPU architecture information is determined. It is determined whether the target vectorized operator involves reading migration data, and based on the determination result, the data processing process corresponding to the execution plan is executed through the target vectorized operator library to obtain the execution result corresponding to the execution plan. Therefore, by converting the target operators involved in the execution plan into corresponding vectorized operators and performing data processing based on the vectorized operators, the computing performance of the heterogeneous environment cluster can be effectively improved, ensuring the efficiency of data processing.

[0106] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the following examples are provided for illustrative purposes:

[0107] As an example, in this embodiment, the program involved in the data processing method is added to the Spark task as a plugin. After the client submits the SQL task, the Driver parses the task and generates an execution plan. Using an operator dictionary, operators that can utilize vectorization are converted into vectorized operators. The execution plan is then distributed to the executor. Before executing the operators, the Task thread on the executor first determines the local CPU architecture. If the local CPU architecture is x86, the underlying operator library is dynamically set to Velox; if the local CPU architecture is ARM, the underlying operator library is dynamically set to OmniOperator. Using the architecture-dependent operator library, the underlying vectorized instruction set is called to read and process local data, and the results are returned to Spark's native operators via JNI for further processing.

[0108] Specifically, during the task initiation phase, the operator library directly reads the raw input data from the hard drive and converts it into a data structure unique to this operator for subsequent calculations. During the shuffle phase, when downstream data pulls upstream shuffle data, it simultaneously determines the local CPU architecture and obtains the CPU architecture of the upstream node. If the architectures are the same, the data is directly pulled for subsequent processing. If the architectures are different, the data from the upstream node is converted into a data structure supported by the local vectorization library during the pulling process as needed, and then further calculations are performed.

[0109] For example, referring to Figure 2, a schematic diagram of the calculation process provided in the embodiment of this application is shown. The specific process may include:

[0110] 1.1: The client submits a SparkSQL task;

[0111] 1.2: SparkDriver parses the SQL and generates an execution plan;

[0112] 1.3: Determine whether the execution plan can utilize vectorized computation features by using a dictionary of vectorized operators;

[0113] 1.4: Determine whether the operator in the execution plan supports vectorization. If all CPU architectures in the cluster support the operator, mark the operator as supported; otherwise, mark the operator as unsupported.

[0114] 1.5: If not supported, the native Spark computing engine will be used directly for computation and the results will be returned.

[0115] 1.6: If supported, convert the relevant operators in the execution plan into vectorized operators;

[0116] 1.7: The execution plan is sent to the SparkExecutor (when the executor starts, it calls the operating system interface to obtain information such as CPU architecture and stores it in the executor's local variables);

[0117] 1.8: The Task thread in the Executor begins executing the assigned operator;

[0118] 1.9: Before actually performing vectorized computation, obtain CPU architecture information from the executor's local variables to dynamically determine the vectorized operator library;

[0119] 1.10: If the CPU is an x86 architecture, use the Velox library; if the CPU is an ARM architecture, use the Omni library.

[0120] 1.11: Determine whether the operator reads shuffle data. If not, unload the operator to the native engine, call the underlying vectorized instruction set to execute the vectorized operator, and return the result.

[0121] 1.12: If it is shuffle data, then determine whether the CPU architecture of the upstream and downstream nodes is the same. If it is the same, the downstream node directly pulls the intermediate shuffle data of the upstream node, performs vectorized calculation, and returns the result.

[0122] 1.13: If the upstream and downstream nodes have different CPU architectures, the data structure is transformed while the downstream node is pulling shuffle data from the upstream node. The vectorized data structure of the upstream node is converted into a vectorized data structure of the downstream node. After the conversion is complete, vectorized calculations are performed, and the results are returned.

[0123] For example, suppose a user queries the following in the client: SQL statement: SELECT sex,count(1)FROM STUDENT WHERE age>14 GROUP BY sex. That is, find all students older than 14 years old from the STUDENT table, group these students by gender, and calculate the number of male and female students respectively. Then, as provided in the above embodiment, the vectorized calculation process can perform lexical and syntactic parsing on the SQL to generate a physical plan tree. The logical representation of the physical plan tree and the corresponding Java code are shown in Figure 3. Specifically, the Java code corresponding to the VectortableScan operator reads the table file line by line, stores it in the collection iterator, and provides it for downstream use; the VectorFilter operator obtains the upstream iterator, scans the data line by line, filters the data with age greater than 14, and groups the data by sex locally, saving it in the local temporary shuffle file; the VectorGroup operator's task pulls the shuffle data from the server where the upstream operator task is located, performs aggregation calculation in local memory, and finally displays the output.

[0124] Referring to Figure 4, a schematic diagram of the data processing provided in this embodiment is shown. During the execution of vectorized calculations, threads Task1 (x86), Task2 (ARM), and Task3 (ARM) can respectively read corresponding data from the STUDENT data table, such as:

[0125] Data read by thread Task1 (x86):

[0126] Zhang Yi, 15, male

[0127] Li Yi, 16, male

[0128] Wang Yi, 13, male

[0129] Yanzi, 15, female

[0130] Data read by thread Task2 (ARM):

[0131] Zhang Er, 16, male

[0132] Li Er, 13, male

[0133] Wang Er, 13, male

[0134] Parrot, 16, female

[0135] Data read by thread Task3 (ARM):

[0136] Zhang San, 13, male

[0137] Li San, 15, male

[0138] Wang San, 15, male

[0139] Eagle, 13, Female

[0140] Next, after each thread reads the corresponding data from the data table, it can perform the corresponding vector calculations to extract data with an age greater than 14. The extracted data would then be:

[0141] Data extracted by thread Task1 (x86):

[0142] Zhang Yi, 15, male

[0143] Li Yi, 16, male

[0144] Yanzi, 15, female

[0145] Data extracted by thread Task2 (ARM):

[0146] Zhang Er, 16, male

[0147] Parrot, 16, female

[0148] Data extracted by thread Task3 (ARM):

[0149] Li San, 15, male

[0150] Wang San, 15, male

[0151] Meanwhile, when migrating data between different computing nodes, the system can determine whether data structure conversion is necessary based on whether the CPU architectures are the same. For example, if Task1 (x86) and Task2 (ARM) have different CPU architectures, then during the data retrieval process, x86ShuffleData needs to be converted to armShuffleData; conversely, the same data structure conversion is performed. When the CPU architectures are the same, data can be retrieved directly. After the data retrieval is complete, the corresponding vectorized calculations are performed, and the corresponding calculation results are returned. For example, the final output result could be: 5 males and 2 females, meaning there are 5 students older than 14 and 2 students older than 14.

[0152] In the above process, the vectorized library to be used can be dynamically determined based on the local architecture information only before the vectorized operator is actually unloaded to the native library for computation. Furthermore, by modifying the Spark shuffle logic, the compatibility of intermediate data structures between operator libraries of different architectures is ensured during transmission. Additionally, a global vectorized operator dictionary is maintained to record vectorized operators that can be supported by all CPU architectures in the cluster. Moreover, vectorized computing technology supports the improvement of computing performance in heterogeneous environment clusters, thereby eliminating the need to focus on the underlying CPU architecture information. By using Spark's vectorized computing features, computing efficiency is improved while reducing the difficulty of resource management in heterogeneous CPU clusters.

[0153] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0154] Referring to Figure 5, a structural block diagram of a data processing device provided in an embodiment of this application is shown. This device is applied to a heterogeneous CPU hybrid deployment cluster, which includes several servers, each server including several computing nodes. Specifically, it may include the following modules:

[0155] The task decomposition module 501 is used to decompose the data processing task in response to receiving the data processing task, and obtain the execution plan corresponding to the data processing task. The execution plan includes a number of execution processes and a target operator corresponding to each execution process.

[0156] The configuration module 502 is used to convert each of the target operators into target vectorized operators, allocate each of the target vectorized operators to the corresponding target computing nodes, and obtain the CPU architecture information corresponding to the target computing nodes to determine the target vectorized operator library that matches the CPU architecture information.

[0157] The execution module 503 is used to determine whether the target vectorization operator involves reading migration data, and to execute the data processing process corresponding to the execution plan through the target vectorization operator library according to the determination result, so as to obtain the execution result corresponding to the execution plan.

[0158] In some feasible implementations, the configuration module 502 is specifically used for:

[0159] Obtain a vectorized operator dictionary for the target operator, wherein the vectorized operator dictionary maintains the vectorized operators supported by the computing node;

[0160] If the target operator is located in the vectorized operator dictionary, and the CPU architecture in the computing node supports the target operator, then the target operator is converted into a target vectorized operator.

[0161] Among some feasible implementation methods are:

[0162] The computation execution module is configured to execute the execution plan using the native computation engine if the target operator is not located in the vectorized operator dictionary and / or the CPU architecture in the computation node does not support the target operator.

[0163] In some feasible implementations, the configuration module 502 is specifically used for:

[0164] If the CPU architecture information is x86 architecture, then determine the first vectorized operator library corresponding to the x86 architecture;

[0165] If the CPU architecture information is ARM architecture, then determine the second vector operator library corresponding to the ARM architecture.

[0166] In some feasible implementations, the target computing node includes a native engine, and the execution module 503 is specifically used for:

[0167] If the judgment result indicates that the target vectorization operator involves reading migration data, then the first CPU architecture information corresponding to the upstream computing node and the second CPU architecture information corresponding to the downstream computing node in the target computing node are obtained. Data conversion is performed according to the first CPU architecture information and the second CPU architecture information. Based on the result of the data conversion, the data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain the execution result corresponding to the execution plan.

[0168] If the judgment result indicates that the target vectorization operator does not involve reading migration data, then the target vectorization operator is unloaded into the native engine of the target computing node, and the underlying vectorization instruction set is called through the vectorization operator library to execute the target vectorization operator and obtain the execution result corresponding to the execution plan.

[0169] In some feasible implementations, the execution module 503 is specifically used for:

[0170] If the first CPU architecture information is different from the second CPU architecture information, then while the downstream computing node is pulling the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node. The vectorized operator library is used to perform vectorized calculations on the downstream vectorized data structure to obtain the execution result corresponding to the execution plan.

[0171] In some feasible implementations, the execution module 503 is specifically used for:

[0172] If the first CPU architecture information is the same as the second CPU architecture information, then the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized and calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

[0173] Each module in the aforementioned processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0174] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0175] In addition, this application also provides an electronic device (not shown), including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described data processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0176] This application also provides a computer-readable storage medium (not shown in the figures) storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0177] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0178] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program code.

[0179] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0182] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0183] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0184] The above provides a detailed description of a data processing method and a data processing apparatus provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method applied to a heterogeneous CPU hybrid deployment cluster, wherein the heterogeneous CPU hybrid deployment cluster includes a plurality of servers, each server including a plurality of computing nodes, the method comprising: In response to receiving a data query task, the data query task is decomposed to obtain an execution plan corresponding to the data query task. The execution plan includes several execution processes and a target operator corresponding to each execution process. Each of the target operators is converted into a target vectorized operator, and each of the target vectorized operators is assigned to the corresponding target computing nodes; Obtain the CPU architecture information corresponding to the target computing node, and determine the target vectorized operator library that matches the CPU architecture information; as well as Determine whether the target vectorization operator involves reading migration data, and based on the determination result, execute the data processing process corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan.

2. The method according to claim 1, wherein, The step of converting each of the target operators into target vectorized operators includes: Obtain a vectorized operator dictionary for the target operator, wherein the vectorized operator dictionary maintains the vectorized operators supported by the computing node; If the target operator is located in the vectorized operator dictionary, and the CPU architecture in the computing node supports the target operator, then the target operator is converted into a target vectorized operator.

3. The method according to claim 2, wherein, Also includes: If the target operator is not in the vectorized operator dictionary, and / or the CPU architecture in the computing node does not support the target operator, then the execution plan is executed using the native computing engine.

4. The method according to claim 1, wherein, The determination of the target vectorized operator library that matches the CPU architecture information includes: If the CPU architecture information is x86 architecture, then determine the first vectorized operator library corresponding to the x86 architecture; If the CPU architecture information is ARM architecture, then determine the second vector operator library corresponding to the ARM architecture.

5. The method according to claim 1, wherein, The target computing node contains a native engine. The step of executing the data processing procedure corresponding to the execution plan through the target vectorized operator library based on the judgment result to obtain the execution result corresponding to the execution plan includes: If the judgment result indicates that the target vectorization operator involves reading migration data, then the first CPU architecture information corresponding to the upstream computing node and the second CPU architecture information corresponding to the downstream computing node in the target computing node are obtained. Data conversion is performed according to the first CPU architecture information and the second CPU architecture information. Based on the result of the data conversion, the data processing process corresponding to the execution plan is executed through the target vectorization operator library to obtain the execution result corresponding to the execution plan. If the judgment result indicates that the target vectorization operator does not involve reading migration data, then the target vectorization operator is unloaded into the native engine of the target computing node, and the underlying vectorization instruction set is called through the vectorization operator library to execute the target vectorization operator and obtain the execution result corresponding to the execution plan.

6. The method according to claim 5, wherein, The step of performing data conversion based on the first CPU architecture information and the second CPU architecture information, and then, based on the converted data, executing the data processing procedure corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan, includes: If the first CPU architecture information is different from the second CPU architecture information, then while the downstream computing node is pulling the migration data of the upstream computing node, the pulled upstream vectorized data structure is converted into a downstream vectorized data structure that matches the second CPU architecture information of the downstream computing node. The vectorized operator library is used to perform vectorized calculations on the downstream vectorized data structure to obtain the execution result corresponding to the execution plan.

7. The method according to claim 5 or 6, wherein, The step of performing data conversion based on the first CPU architecture information and the second CPU architecture information, and then, based on the converted data, executing the data processing procedure corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan, includes: If the first CPU architecture information is the same as the second CPU architecture information, then the target data of the upstream computing node is pulled through the downstream computing node, and the target data is vectorized and calculated through the vectorized operator library to obtain the execution result corresponding to the execution plan.

8. A data processing apparatus applied to a heterogeneous CPU hybrid deployment cluster, the heterogeneous CPU hybrid deployment cluster including a plurality of servers, each server including a plurality of computing nodes, the apparatus comprising: The task decomposition module is used to decompose the data query task in response to receiving the data query task, and obtain the execution plan corresponding to the data query task. The execution plan includes several execution processes and a target operator corresponding to each execution process. The configuration module is used to convert each of the target operators into target vectorized operators, allocate each of the target vectorized operators to the corresponding target computing nodes, and obtain the CPU architecture information corresponding to the target computing nodes to determine the target vectorized operator library that matches the CPU architecture information. The execution module is used to determine whether the target vectorization operator involves reading migration data, and based on the determination result, to execute the data processing process corresponding to the execution plan through the target vectorization operator library to obtain the execution result corresponding to the execution plan.

9. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-7.

Images