Data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of computers, the method comprises the following steps: determining a to-be-executed physical plan, and calculating each target node associated with the physical plan in a cluster; judging an computable mode corresponding to each target node; converting the physical plan into a target code suitable for vectorization calculation under the condition that the operational mode corresponding to the target node meets the vectorization calculation condition; and running the corresponding target code based on the target node. Therefore, vectorization calculation can be performed on the physical plan based on the target node meeting the condition in the calculation cluster, data processing is remarkably accelerated, task time consumption is reduced, calculation efficiency and the resource utilization rate are improved, and good universality is achieved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more particularly to a data processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the rapid development of technologies such as artificial intelligence, cloud computing, and the Internet of Things, the generation and accumulation of data in the big data era has exploded in scale, dimension, and speed. Large-scale data processing faces major challenges, and distributed computing frameworks have become key technologies. Spark distributed clusters, as a leader, have the advantages of high throughput, strong fault tolerance, and superior performance.

[0003] However, many distributed clusters, such as Spark, do not support vectorized computing, have low parallel granularity in data processing, and perform poorly in large-scale parallel computing tasks. Summary of the Invention

[0004] Embodiments of the present disclosure provide a data processing method, device, electronic device, and computer-readable storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a data processing method, the method comprising:

[0006] Determining a physical plan to be executed and target nodes associated with the physical plan in the computing cluster;

[0007] Determine the operable mode corresponding to each target node;

[0008] If the operable mode corresponding to the target node meets the conditions for vectorized computing, converting the physical plan into target code suitable for vectorized computing;

[0009] The corresponding target code is run based on the target node.

[0010] In a second aspect, an embodiment of the present disclosure further provides a data processing device, the device comprising:

[0011] A first determination module is used to determine a physical plan to be executed and each target node associated with the physical plan in the computing cluster;

[0012] A judgment module, configured to judge the operable mode corresponding to each target node;

[0013] a conversion module, configured to convert the physical plan into target code suitable for vectorized computing if the operable mode corresponding to the target node meets the conditions for vectorized computing;

[0014] A running module is used to run the corresponding target code based on the target node.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the above-mentioned data processing method are implemented.

[0016] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned data processing method are implemented.

[0017] In a fifth aspect, embodiments of the present disclosure further provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in the embodiments of the present disclosure.

[0018] In summary, the disclosed embodiment first determines the physical plan to be executed and the target nodes associated with the physical plan in the computing cluster. It then determines the operable mode corresponding to each target node. If the operable mode corresponding to the target node meets the conditions for vectorized computing, the physical plan is converted into target code suitable for vectorized computing, and the corresponding target code is then run based on the target node. This allows vectorized computing of the physical plan based on eligible target nodes in the computing cluster, significantly accelerating data processing and reducing task time, thereby improving computing efficiency and resource utilization, and possessing excellent versatility.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a data processing method provided by the first embodiment of the present disclosure;

[0022] Figure 2is a flowchart of a data processing method provided by the second embodiment of the present disclosure;

[0023] Figure 3 is a structural diagram of a data processing device provided by an embodiment of the present disclosure;

[0024] Figure 4 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Some embodiments of the present disclosure will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications and equivalents of the methods, devices and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but may be changed as becomes apparent after understanding the present disclosure, except for operations that must be performed in a specific order. In addition, for the sake of clarity and brevity, descriptions of features known in the art may be omitted.

[0026] The embodiments described in the following examples of the present disclosure do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0027] It should be noted that the execution entity of the data processing method of this embodiment may be a distributed computing system, which is not limited here.

[0028] It should be noted that due to the limitations of Spark's computing model, the minimum granularity of its parallel computing can only reach the task level. For computationally intensive tasks, Spark cannot use the CPU SIMD instruction set for batch processing and can only iterate row by row, which greatly increases the computing overhead, resulting in low resource utilization and long task execution time. In addition, Spark's architecture and execution engine are mainly optimized for the CPU and lack native support and adaptation for hardware accelerators such as GPUs, which limits Spark's performance in large-scale parallel computing tasks. In the embodiments of the present disclosure, a data processing method is proposed to solve the problems that may exist in the above background description.

[0029] In the embodiments of the present disclosure, the data processing method will be described using a "distributed computing system" as the execution subject, without limitation. It should be noted that the order of description of the following embodiments does not limit the priority order of the embodiments.

[0030] Figure 1It is a flowchart of a data processing method provided according to the first embodiment of the present disclosure.

[0031] like Figure 1 As shown, the method includes:

[0032] Step 101: Determine a physical plan to be executed and each target node associated with the physical plan in a computing cluster.

[0033] The physical plan to be executed can be a detailed plan of converting a logical query plan into a specific executable operation sequence when the database system processes a user query, such as how to efficiently execute the query task in the actual computer hardware and software environment.

[0034] The computing cluster may be a distributed computing system deployed in the form of a cluster for rapidly processing large-scale data.

[0035] It's important to note that in database systems, an optimized logical plan typically exists. This plan describes the logical operations of a query, such as selection, projection, and join, but doesn't cover the specific physical implementation details. Determining the physical plan involves converting the logical plan into a specific sequence of operations that can actually be executed on a computing cluster.

[0036] For example, in a distributed computing cluster, data may be stored across different nodes. The physical plan determines the data read method and order based on the data's storage location to reduce data transmission overhead. For example, if a table's data is mostly stored on node A, operations involving that table may be preferentially executed on node A.

[0037] It is understandable that in order to determine the target node associated with the physical plan, it is necessary to understand the attribute information of each node, such as:

[0038] Computing resource usage information: such as CPU usage, number of idle cores, etc. Computation-intensive tasks can be preferentially assigned to nodes with sufficient computing resources.

[0039] Memory resource usage information: includes total memory, used memory, and available memory. For operations that require a large amount of memory, select a node with abundant memory resources.

[0040] Network bandwidth: Data transmission between nodes needs to be carried out through the network. Nodes with large network bandwidth have advantages in data transmission and are suitable for tasks that require frequent data interaction.

[0041] Task load information: Understand the number and types of tasks currently being executed by the node to avoid assigning new tasks to overloaded nodes.

[0042] Specifically, tasks can be assigned to appropriate target nodes based on the requirements of each operation in the physical plan and the node's attributes. For example, for data read operations, nodes storing the corresponding data can be selected as the target node; for computation operations, nodes with sufficient computing resources can be selected as the target node. Furthermore, inter-node collaboration and data transmission must be considered to ensure efficient execution of the entire physical plan.

[0043] Step 102: Determine the operable mode corresponding to each target node.

[0044] The operable mode indicates the operation mode that any computing node can currently adopt, such as vectorized computing and scalar computing, and there is no restriction on this.

[0045] The vectorized operations include GPU (Graphics Processing Unit) operations, CPU SIMD operations, etc., which are not limited here.

[0046] Among them, GPU computing can be parallel computing using a graphics processor, which is suitable for large-scale data parallel processing.

[0047] The CPU's (Central Processing Unit) SIMD (Single Instruction, Multiple Data) operation is a technology for implementing parallel computing on the CPU. It allows a single instruction to perform the same operation on multiple data elements simultaneously, processing multiple data elements with a single instruction execution, thereby improving data processing efficiency.

[0048] For example, suppose there are four target nodes, A, B, C, and D. Target node A's compatibility mode supports only GPU computing. Target node B's compatibility mode does not support GPU computing but supports CPU SIMD computing. Target node C's compatibility mode supports both GPU and CPU SIMD computing. Target node D's compatibility mode is scalar computing.

[0049] Step 103 : When the operable mode corresponding to the target node meets the conditions for vectorized computing, the physical plan is converted into target code suitable for vectorized computing.

[0050] Optionally, when the computational mode of the target node includes GPU computing, it can be determined that the target node meets the conditions for vectorized computing.

[0051] For example, if the current computational mode of the target node includes GPU computation but does not support CPU SIMD computation, it can be determined that the target node can implement vectorized computation through GPU computation and meets the conditions for vectorized computation.

[0052] Alternatively, if the current computational mode of the target node includes GPU computation but also supports CPU SIMD computation, it can be determined that the target node can implement vectorized computation through GPU computation or CPU SIMD computation, thus meeting the conditions for vectorized computation.

[0053] Optionally, when the operable mode of the target node includes CPU SIMD operation, it can be determined that the target node meets the conditions for vectorized computing.

[0054] For example, if the current computational mode of the target node includes CPU SIMD computation, but also supports GPU computation, it can be determined that the target node can implement vectorized computation through GPU computation or CPU SIMD computation, thus meeting the conditions for vectorized computation.

[0055] Alternatively, if the current operable mode of the target node includes CPU SIMD operations but does not support GPU operations, it can be determined that the target node can implement vectorized operations through CPU SIMD operations and meet the conditions for vectorized computing.

[0056] The target code may be a code suitable for vectorized operations obtained by converting a physical plan.

[0057] Specifically, the most suitable target code generation method can be selected based on the hardware capabilities of the target node. For example, if the target node supports GPU computing, the physical plan can be converted into target code suitable for GPU computing. GPUs are suitable for highly parallel tasks (such as matrix operations and large-scale data filtering).

[0058] Optionally, when the target node's operable mode includes GPU computing, the physical plan may be converted into target code suitable for GPU computing.

[0059] Alternatively, you can use a GPU programming framework (such as CUDA, OpenCL) to generate target code, transfer data in batches to the GPU memory, and use the parallel computing capabilities of the GPU to execute tasks.

[0060] Optionally, when the target node's operable mode includes CPU SIMD operations but does not include GPU operations, the physical plan is converted into target code suitable for CPU SIMD operations.

[0061] CPU SIMD operations are a type of vectorized computing that leverages the CPU's Single Instruction, Multiple Data (SIMD) feature to process multiple data elements in a single operation, thereby improving computing efficiency. This involves converting a physical plan into target code suitable for vectorized computing. This means that a physical plan that might have originally processed data row by row is reorganized and compiled into code that can process data in batches in vector form.

[0062] Specifically, by analyzing the operational steps and data processing logic in the physical plan, we can determine which operations can be performed using vectorization. For example, operations such as batch addition, subtraction, multiplication, and division of array elements, as well as batch comparisons, are generally suitable for vectorization. To facilitate vectorized calculations, data can be stored using an appropriate layout. For example, data can be organized into contiguous memory blocks so that SIMD instructions can efficiently access and process the data. Based on the analysis results and the adjusted data layout, the physical plan can then be converted into target code using a specific compiler or code generation tool. This target code can call the underlying SIMD instruction set functions to implement batch data processing.

[0063] In summary, the optimal computing mode can be dynamically selected according to the hardware capabilities of the target node (GPU or CPU SIMD), and the physical plan can be converted into a suitable target code, which significantly improves the execution efficiency of the query and fully utilizes the hardware resources.

[0064] In the disclosed embodiment, code conversion can be achieved through the Spark plug-in mechanism. Users only need to simply add a Jar package to achieve automatic acceleration of calculations without spending a lot of effort to modify online business logic.

[0065] Step 104: Run the corresponding target code based on the target node.

[0066] The target node is the node in the computing cluster responsible for executing specific computing tasks.

[0067] Specifically, the generated target code can be distributed to the selected target node. The code file can be copied to the designated directory of the target node through network transmission or other means. The target node must be equipped with the necessary environment to run the target code, including installing the necessary dependent libraries and configuring the correct operating parameters.

[0068] Specifically, the target code can be started on the target node. The target node's operating system will call the corresponding interpreter or compiler to run the code, processing data in a vectorized manner to improve computing efficiency.

[0069] It can be understood that, since the target code is a code for vectorized operation, running the target code through the target node is to perform vectorized operation.

[0070] Optionally, when the operability mode of the target node does not meet the conditions for vectorized calculation, the physical plan is scalarized and executed based on the target node.

[0071] Among them, scalarization is a calculation method that processes single data elements one by one at a time, and each operation acts on only one data element.

[0072] It should be noted that in actual big data computing scenarios, different computing clusters have different hardware configurations, and not all target nodes have the hardware conditions to support vectorized computing. For example, some older servers may not have CPUs that support the SIMD instruction set or have insufficient GPU resources. If the target node's operational mode does not meet the conditions for vectorized computing, vectorized computing cannot be performed.

[0073] Therefore, the physical plan can be scalarized based on the target node. This ensures that even if the cluster hardware doesn't support vectorized computation, the computation task can still be successfully executed. This provides a fallback solution for big data computation, preventing the entire task from failing due to partial node support for vectorized computation, and ensuring system stability and reliability. This preserves Spark's scalar execution functionality and provides an automatic fallback mechanism, ensuring successful computation even if the cluster doesn't support the corresponding hardware. This is achieved through the Spark plugin mechanism, making user integration easy.

[0074] In summary, the disclosed embodiment first determines the physical plan to be executed and the target nodes associated with the physical plan in the computing cluster. It then determines the operable mode corresponding to each target node. If the operable mode corresponding to the target node meets the conditions for vectorized computing, the physical plan is converted into target code suitable for vectorized computing, and the corresponding target code is then run based on the target node. This allows vectorized computing of the physical plan based on eligible target nodes in the computing cluster, significantly accelerating data processing and reducing task time, thereby improving computing efficiency and resource utilization, and possessing excellent versatility.

[0075] Figure 2 It is a flowchart of a data processing method provided according to the second embodiment of the present disclosure.

[0076] like Figure 2 As shown, the method includes:

[0077] Step 201: Receive a submitted structured query language SQL character string.

[0078] The SQL query string is a text string written in SQL (Structured Query Language), which is used to send a request to a database management system (DBMS) to retrieve, insert, update, or delete data in the database.

[0079] Optionally, if the distributed computing system is Spark, the user can submit an SQL query string through a Spark SQL interface, such as the Spark SQL CLI (Spark SQL Command-Line Interface).

[0080] Optionally, an SQL query string may be submitted through a Java Database Connectivity / Open Database Connectivity Interface (JDBC / ODBC interface) or a DataFrame Application Programming Interface (DataFrame API, DataFrame Application Programming Interface).

[0081] The SQL string can be a simple query (such as SELECT * FROM table, which selects all columns in a table) or a complex query, such as one containing join operations (JOIN), grouping operations (GROUP BY), sub-queries, etc. After receiving the SQL string, Spark can pass it as input to the SQL parser.

[0082] As an example, the SQL string is "SELECT * FROM employees". This query uses "SELECT *" to select all columns of the employees table. "FROM employees" indicates that the data comes from the "employees table" and is not limited here.

[0083] Step 202: Parse the SQL string to obtain an abstract syntax tree.

[0084] The abstract syntax tree is a tree structure, and each node represents a component in an SQL statement, such as SELECT, FROM, WHERE, etc.

[0085] Optionally, an SQL parser may be used to parse the SQL string. The parser may convert the SQL string into an Abstract Syntax Tree (AST) based on grammatical rules.

[0086] Step 203: Check the abstract syntax tree to obtain a check result.

[0087] Optionally, the parsed abstract syntax tree can be passed to a parser first. The main task of the parser is to perform semantic analysis on the abstract syntax tree, such as checking whether table names and column names exist, checking whether table and column data types match, and checking parsing functions and expressions, which are not limited here.

[0088] Optionally, you can bind symbols in the abstract syntax tree (such as table names and column names) to actual metadata (such as table information in the catalog), and then check whether the SQL statement complies with semantic rules (such as whether the GROUP BY columns are in the SELECT and whether aggregate functions are used correctly).

[0089] It should be noted that for different inspection types, the corresponding inspection results are usually different.

[0090] Step 204: Based on the checking result, obtain an initial logical plan corresponding to the abstract syntax tree.

[0091] Optionally, when the check result shows that the abstract syntax tree has no syntax errors, the table information and column information associated with the abstract syntax tree can be determined, and then the target verification information associated with the table information and column information can be determined, and then the target verification information can be verified to obtain a verification result. Then, when the verification result is passed, an initial logical plan corresponding to the abstract syntax tree is generated.

[0092] Specifically, the generated abstract syntax tree can be checked to see if it complies with the grammatical rules of the SQL language. For example, it can check whether keywords are used correctly, brackets are matched, and the statement structure is complete. If the check result indicates that the abstract syntax tree contains syntax errors, an error message can be returned to the user, prompting them to modify the SQL query statement.

[0093] Optionally, when the check result shows that the abstract syntax tree has no syntax errors, table information and column information associated with the abstract syntax tree may be determined.

[0094] For example, for the SQL statement "SELECT column1, column2 FROM table1 JOIN table2 ON table1.id = table2.id", the table information involved is "table1" and "table2".

[0095] Column information includes the column names used in the query and the tables to which these columns belong. In the above example, the column information includes "table1.column1", "table1.column2", "table1.id", and "table2.id".

[0096] Furthermore, target verification information associated with the table information and column information may be determined.

[0097] The target verification information may be metadata corresponding to the table information and column information, and is used to verify whether the table involved in the abstract syntax tree actually exists in the database.

[0098] For example, when checking for the existence of a column, you can check whether each column belongs to its specified table. For example, to verify whether "column1" in "table1.column1" is indeed a column of the "table1" table.

[0099] You can also determine whether the column's data type is compatible with how it is used in the query. For example, if a column is used for numeric operations in a query, the column's data type should be a numeric type.

[0100] Optionally, after the validation is complete, you will receive a validation result, which may be passed or failed. If all table and column names exist and the field types are correct, the validation result is passed. Conversely, if there are problems such as non-existent tables, columns, or data types, the validation result is failed.

[0101] The initial logical plan is an abstract representation of a SQL query, describing the execution steps and sequence of operations without specifying the physical implementation details. For example, the logical plan might specify that table joins should be performed first, followed by filtering and projection operations. If validation passes, an initial logical plan corresponding to the abstract syntax tree (AST) is generated based on the AST and associated metadata. This initial logical plan serves as the basis for subsequent query optimization and physical execution.

[0102] Step 205 : Optimize the initial logical plan based on preset optimization rules to obtain an optimized target logical plan.

[0103] It is understandable that the initial logical plan is a preliminary planning of the SQL query operation steps and does not take into account how to execute the query in the most efficient manner.

[0104] In the disclosed embodiment, the optimizer can analyze the initial logical plan, combine the database metadata (such as table statistics, index information, etc.) and system resource conditions, and use a series of optimization strategies to adjust and improve it to generate a target logical plan with higher execution efficiency.

[0105] The preset optimization rules may include predicate pushdown, projection clipping, etc., which are not limited here.

[0106] Predicates refer to filter conditions in SQL queries. Predicate pushdown applies filter conditions as early as possible during the data reading phase. In the initial logical plan, filtering operations may be performed after operations such as table joins, but this results in processing a large amount of unnecessary data. Predicate pushdown applies filter conditions during data reading, reducing the amount of data involved in subsequent operations and thus lowering computational overhead.

[0107] Projection refers to selecting specific columns in a SQL query, while projection pruning removes unnecessary columns from the query. The initial logical plan might read all table columns, but the query actually requires only a subset. Projection pruning reduces data transmission and processing, improving query efficiency by only reading the necessary columns.

[0108] Specifically, the initial logical plan can be analyzed and adjusted using preset optimization rules, such as predicate pushdown and projection clipping. During this process, the optimizer considers factors such as the execution order of different operations and data access paths (such as whether indexes are used), weighing various possible execution options and ultimately generating a more optimized target logical plan. This optimized target logical plan can complete query tasks with less resource consumption and in a shorter time, improving the overall performance of the database system.

[0109] Step 206: Determine each initial node in the computing cluster that is associated with the target logical plan.

[0110] The target logical plan describes the abstract steps and operation sequence of query execution, but actual execution needs to be implemented on specific nodes in the computing cluster.

[0111] It should be noted that, in order to determine the initial nodes associated with the target logical plan, it is necessary to find out which nodes in the computing cluster may participate in the execution process of the target logical plan.

[0112] For example, the initial node can be determined based on the location of data storage (if distributed storage is used, different data blocks may be stored on different nodes), the computing power of the node, etc.

[0113] For example, consider a distributed database cluster with multiple nodes, and a target logical plan involves querying and processing a large data table. The table is distributed across some nodes, and the nodes storing the relevant data are identified as the initial nodes associated with the target logical plan because the query needs to read data from these nodes for operation.

[0114] Step 207: Convert the target logical plan into a physical plan to be executed, and determine each target node from each initial node.

[0115] It's important to note that the target logical plan focuses on describing the query's logical operations, such as joins, filters, and projections, and typically doesn't cover the specific physical implementation details. The physical plan, on the other hand, considers how these operations are executed on the actual nodes in the computing cluster, including data transmission methods, the specific algorithms used, and resource allocation.

[0116] Optionally, you can first determine the attribute information corresponding to each initial node, and the attribute information includes at least one of the following: computing resource usage information, memory resource usage information, task execution information, or load information. Then, based on the attribute information corresponding to each initial node, convert the target logical plan into a physical plan to be executed, and determine each target node from each initial node.

[0117] Among them, the attribute information of the node is important for generating an efficient physical plan and selecting a suitable target node. The attribute information of the node can include computing resource usage information, memory resource usage information, task execution information or load information, etc., which is not limited here.

[0118] The computing resource usage information may include the CPU usage of the node, the number of available computing cores, etc. For example, if the CPU of a node is already in a high load state, then it may be necessary to be cautious when allocating tasks.

[0119] Memory resource usage information may include the total amount of memory, used memory, and available memory of the node. If an operation requires a large amount of memory, you need to select a node with sufficient memory resources.

[0120] The task execution information may include the number of tasks currently being executed by the node, the priority of the tasks, etc., to avoid allocating new tasks to nodes that are already overloaded with tasks.

[0121] The load information may be an overall load indicator obtained by comprehensively considering factors such as computing resources, memory resources, and task execution status, and is used to evaluate the busyness of the node.

[0122] As an example, the target logical plan can be converted into a physical plan based on the attribute information of each initial node. For example, if an operation requires a large amount of computing resources, nodes with good computing resource usage information (such as idle CPUs and a large number of computing cores) will be preferentially selected as the target node. For operations requiring a large amount of memory, nodes with sufficient memory resources will be selected. At the same time, factors such as the data transmission cost between nodes are also considered to ensure the execution efficiency of the physical plan.

[0123] For example, suppose the target logical plan includes a data aggregation operation that requires significant computing resources. Among the initial nodes, initial node A has low CPU utilization and multiple idle cores, while initial node B's CPU is already fully utilized. Therefore, when converting the target logical plan to a physical plan, the data aggregation operation is preferentially assigned to initial node A. In other words, initial node A is identified as one of the target nodes.

[0124] Step 208: Determine the operable mode corresponding to each target node.

[0125] Step 209 : If the operable mode corresponding to the target node meets the conditions for vectorized computing, convert the physical plan into target code suitable for vectorized computing.

[0126] Step 210: Run the corresponding target code based on the target node.

[0127] It should be noted that the specific implementation of steps 208 - 210 may refer to the above embodiment and will not be described in detail here.

[0128] In the embodiment of the present disclosure, a submitted structured query language SQL string is first received, the SQL string is then parsed to obtain an abstract syntax tree, the abstract syntax tree is then checked to obtain a check result, an initial logical plan corresponding to the abstract syntax tree is obtained based on the check result, the initial logical plan is then optimized based on a preset optimization rule to obtain an optimized target logical plan, each initial node associated with the target logical plan in the computing cluster is then determined, the target logical plan is then converted into a physical plan to be executed, and each target node is determined from each initial node, the operable mode corresponding to each target node is determined, and if the operable mode corresponding to the target node meets the conditions for vectorized calculation, the physical plan is converted into a target code suitable for vectorized calculation, and finally the corresponding target code is run based on the target node. In this way, query execution efficiency can be significantly improved, resource consumption can be reduced, system scalability can be improved, complex queries and multiple data sources can be supported, development efficiency and user experience can be improved, and large-scale data processing needs can be adapted. Execution paths can be flexibly selected based on cluster hardware availability. If supported, Spark can leverage the parallel computing capabilities of multiple GPU cores or utilize CPU SIMD instructions for vectorized execution. This significantly accelerates data processing, reduces task duration, and improves computational efficiency and resource utilization. Spark's vectorized execution leverages advanced hardware in a variety of modern computers (GPUs and CPU SIMD), enabling data parallel computing at a finer granularity (register level) to accelerate computation and improve efficiency. With multiple execution paths, users can flexibly select based on their specific needs.

[0129] To facilitate better implementation of the data processing method disclosed herein, the present disclosure further provides a data processing device based on the aforementioned data processing method. The meanings of the terms herein are the same as those in the aforementioned data processing method, and specific implementation details can be found in the description of the method embodiment.

[0130] See also Figure 3 , Figure 3 : is a schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure. The data processing device 300 includes:

[0131] A first determination module 310 is configured to determine a physical plan to be executed and target nodes associated with the physical plan in a computing cluster;

[0132] A determination module 320 is configured to determine an operable mode corresponding to each target node;

[0133] A conversion module 330 is configured to convert the physical plan into target code suitable for vectorized computing if the operable mode corresponding to the target node meets the conditions for vectorized computing;

[0134] The running module 340 is used to run the corresponding target code based on the target node.

[0135] Optionally, the device further includes:

[0136] A second determining module is configured to determine, when the target node's operable mode includes GPU computing, whether the target node meets a condition for vectorized computing;

[0137] The third determining module is configured to determine whether the target node meets the vectorized computing condition when the operable mode of the target node includes CPU SIMD computing.

[0138] Optionally, the conversion module is specifically configured to:

[0139] When the target node's operable mode includes GPU computing, converting the physical plan into target code suitable for GPU computing;

[0140] or,

[0141] When the operable mode of the target node includes CPU SIMD operation but does not include GPU operation, the physical plan is converted into target code suitable for CPU SIMD operation.

[0142] Optionally, the device further includes:

[0143] An execution module is configured to perform scalar execution on the physical plan based on the target node when the operable mode of the target node does not meet the conditions for vectorized calculation.

[0144] Optionally, the first determining module includes:

[0145] An optimization unit, configured to optimize the initial logical plan based on preset optimization rules to obtain an optimized target logical plan;

[0146] A first determining unit, configured to determine each initial node associated with the target logical plan in the computing cluster;

[0147] The second determining unit is configured to convert the target logical plan into a physical plan to be executed, and determine each target node from each initial node.

[0148] Optionally, the optimization unit also includes:

[0149] A receiving subunit, configured to receive a submitted structured query language SQL character string;

[0150] A parsing subunit, configured to parse the SQL string to obtain an abstract syntax tree;

[0151] A checking subunit, configured to check the abstract syntax tree to obtain a checking result;

[0152] The acquisition subunit is configured to acquire an initial logical plan corresponding to the abstract syntax tree based on the checking result.

[0153] Optionally, get subunits, specifically for:

[0154] If the checking result shows that the abstract syntax tree has no syntax errors, determining table information and column information associated with the abstract syntax tree;

[0155] determining target verification information associated with the table information and column information;

[0156] Verifying the target verification information to obtain a verification result;

[0157] If the verification result is passed, an initial logical plan corresponding to the abstract syntax tree is generated.

[0158] Optionally, the second determining unit is specifically configured to:

[0159] Determining attribute information corresponding to each of the initial nodes, the attribute information including at least one of the following: computing resource usage information, memory resource usage information, task execution information, or load information;

[0160] Based on the attribute information corresponding to each of the initial nodes, the target logical plan is converted into a physical plan to be executed, and each target node is determined from each of the initial nodes.

[0161] The disclosed embodiment first determines the physical plan to be executed and the target nodes associated with the physical plan in the computing cluster. It then determines the operable mode corresponding to each target node. If the operable mode corresponding to the target node meets the requirements for vectorized computing, the physical plan is converted into target code suitable for vectorized computing. The target code is then run based on the target node. This allows vectorized computing of the physical plan based on the target nodes that meet the requirements in the computing cluster, significantly accelerating data processing and reducing task time, thereby improving computing efficiency and resource utilization, and possessing excellent versatility.

[0162] In addition, the present disclosure also provides an electronic device, such as Figure 4 , which shows a schematic structural diagram of the electronic device involved in the present disclosure, specifically:

[0163] The electronic device may include one or more processing core processors 401, one or more computer-readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will understand that Figure 4 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0164] Processor 401 is the control center of the electronic device, connecting the various parts of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 402 and accessing data stored in memory 402, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, processor 401 may include one or more processing cores; preferably, processor 401 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0165] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0166] The electronic device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power supply device debugging circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0167] The electronic device may further include an input unit 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0168] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby implementing the steps of any data processing method provided in the embodiments of the present disclosure.

[0169] The disclosed embodiment first determines the physical plan to be executed and the target nodes associated with the physical plan in the computing cluster. It then determines the operable mode corresponding to each target node. If the operable mode corresponding to the target node meets the requirements for vectorized computing, the physical plan is converted into target code suitable for vectorized computing. The target code is then run based on the target node. This allows vectorized computing of the physical plan based on the target nodes that meet the requirements in the computing cluster, significantly accelerating data processing and reducing task time, thereby improving computing efficiency and resource utilization, and possessing excellent versatility.

[0170] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0171] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0172] To this end, the present disclosure provides a computer-readable storage medium having a computer program stored thereon. The computer program can be loaded by a processor to execute the steps in any data processing method provided by the present disclosure.

[0173] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0174] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0175] Since the instructions stored in the computer-readable storage medium can execute the steps in any data processing method provided in the present disclosure, the beneficial effects that can be achieved by any data processing method provided in the present disclosure can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0176] The above is a detailed introduction to a data processing method, device, electronic device and computer-readable storage medium provided by the present disclosure. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A data processing method, characterized in that: include: Determining a physical plan to be executed and target nodes associated with the physical plan in the computing cluster; Determine the operable mode corresponding to each target node; If the operable mode corresponding to the target node meets the conditions for vectorized computing, converting the physical plan into target code suitable for vectorized computing; The corresponding target code is run based on the target node.

2. The method according to claim 1, characterized in that Also includes: In a case where the operable mode of the target node includes graphics processing unit (GPU) operation, determining that the target node meets a condition for vectorized computing; or, In a case where the operable mode of the target node includes CPU SIMD operation, it is determined that the target node meets the conditions for vectorized computing.

3. The method according to claim 2, characterized in that The step of converting the physical plan into target code suitable for vectorized calculation comprises: When the target node's operable mode includes GPU computing, converting the physical plan into target code suitable for GPU computing; or, When the operable mode of the target node includes CPU SIMD operation but does not include GPU operation, the physical plan is converted into target code suitable for CPU SIMD operation.

4. The method according to claim 1, wherein Also includes: When the operable mode of the target node does not meet the conditions for vectorized calculation, the physical plan is scalar-executed based on the target node.

5. The method according to claim 1, characterized in that The determining of a physical plan to be executed and each target node associated with the physical plan in the computing cluster includes: Based on the preset optimization rules, the initial logical plan is optimized to obtain the optimized target logical plan; Determining each initial node in the computing cluster that is associated with the target logical plan; The target logical plan is converted into a physical plan to be executed, and each target node is determined from each initial node.

6. The method according to claim 5, characterized in that Before optimizing the initial logical plan based on the preset optimization rules to obtain the optimized target logical plan, the method further includes: Receive the submitted structured query language SQL string; Parsing the SQL string to obtain an abstract syntax tree; Checking the abstract syntax tree to obtain a check result; Based on the checking result, an initial logic plan corresponding to the abstract syntax tree is obtained.

7. The method according to claim 6, characterized in that The obtaining, based on the checking result, an initial logical plan corresponding to the abstract syntax tree, includes: If the checking result shows that the abstract syntax tree has no syntax errors, determining table information and column information associated with the abstract syntax tree; determining target verification information associated with the table information and column information; Verifying the target verification information to obtain a verification result; If the verification result is passed, an initial logical plan corresponding to the abstract syntax tree is generated.

8. The method according to claim 5, characterized in that The converting the target logical plan into a physical plan to be executed and determining each target node from each initial node includes: Determining attribute information corresponding to each of the initial nodes, the attribute information including at least one of the following: computing resource usage information, memory resource usage information, task execution information, or load information; Based on the attribute information corresponding to each of the initial nodes, the target logical plan is converted into a physical plan to be executed, and each target node is determined from each of the initial nodes.

9. A data processing device, characterized in that: include: A first determination module is used to determine a physical plan to be executed and each target node associated with the physical plan in the computing cluster; A judgment module, configured to judge the operable mode corresponding to each target node; a conversion module, configured to convert the physical plan into target code suitable for vectorized computing if the operable mode corresponding to the target node meets the conditions for vectorized computing; A running module is used to run the corresponding target code based on the target node.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images