Data processing methods, electronic devices and computer storage media

Abstract

This application provides a data processing method, an electronic device, and a computer storage medium. The data processing method includes: determining multiple computation stages of a corresponding data query task in a distributed database system based on the parsing results of a data query statement; creating corresponding task execution operators for each of the multiple computation stages based on the computation tasks corresponding to the multiple computation stages and the degree of parallelism of the computation tasks; wherein, in the computation stages corresponding to multi-parallelism computation tasks, a task execution operator based on inter-process communication is used; in the computation stages corresponding to single-parallelism computation tasks, a task execution operator based on thread communication is used; and the data query task is executed using the task execution operators corresponding to the multiple computation stages. This application embodiment can effectively save resource consumption for data query tasks based on a distributed database system and reduce resource waste.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data processing method, electronic device, and computer storage medium based on a distributed database system. Background Technology

[0002] Distributed database systems are a product of the combination of database technology and network technology. They are characterized by abundant hardware resources, high processing performance, and the ability to meet the needs of real-time big data analysis. Therefore, distributed database systems are widely used in many industries and fields.

[0003] Currently, data query tasks based on distributed database systems typically employ a phased collaborative approach to complete the overall query statement. In this process, task interaction between multiple distributed nodes is a crucial link connecting each stage and is fundamental to fully utilizing distributed hardware resources. However, existing distributed database systems may experience significant hardware resource consumption due to various reasons when completing data query tasks. For example, high-frequency computational data interaction can lead to excessive memory consumption, and hash shuffling operations during the computation phase can result in a large number of network communication connections.

[0004] Therefore, how to save resources for data query tasks based on distributed database systems and reduce resource waste has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide a data processing scheme based on a distributed database system to at least partially solve the above problems.

[0006] According to a first aspect of the embodiments of this application, a data processing method is provided, comprising: determining multiple computation stages of a corresponding data query task in a distributed database system based on the parsing result of a data query statement; creating corresponding task execution operators for each of the multiple computation stages based on the computation tasks corresponding to the multiple computation stages and the parallelism of the computation tasks; wherein, in the computation stages corresponding to multi-parallelism computation tasks, a task execution operator based on inter-process communication is used; in the computation stages corresponding to single-parallelism computation tasks, a task execution operator based on thread communication is used; and the data query task is executed using the task execution operators corresponding to the multiple computation stages.

[0007] According to a second aspect of the embodiments of this application, another data processing method is provided, comprising: receiving a data query statement for querying data from stored cloud logs; logically parsing the data query statement and determining, based on the parsing result, multiple computation stages of the corresponding data query task in the distributed database system storing the cloud logs; creating corresponding task execution operators for each of the multiple computation stages based on the computation tasks corresponding to the multiple computation stages and the parallelism of the computation tasks; wherein, in the computation stages corresponding to multi-parallelism computation tasks, a task execution operator based on inter-process communication is used; in the computation stages corresponding to single-parallelism computation tasks, a task execution operator based on thread communication is used; and executing the data query task for the cloud logs using the task execution operators corresponding to the multiple computation stages.

[0008] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first or second aspect.

[0009] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in the first or second aspect.

[0010] According to the solution provided in the embodiments of this application, when performing data queries based on a distributed database system, different task execution operators are constructed for the computation tasks and their parallelism at different computation stages. Data query tasks are then executed based on these operators. Specifically, if a computation stage requires multi-parallelism computation tasks, a task execution operator based on inter-process communication is used. This type of operator uses inter-process communication to achieve data interaction between upstream and downstream computation tasks, such as data retrieval, caching, and distribution. Compared to the traditional method of achieving data interaction between upstream and downstream computation tasks through inter-thread communication, this significantly reduces the number of network communication links, thereby reducing hardware resource consumption. Conversely, if a computation stage requires single-parallelism computation tasks, a task execution operator based on thread communication is used. This type of operator can directly retrieve data for corresponding computation without caching or distribution operations. On the one hand, in the case of single-parallelism, it ensures the smooth execution of the computation task; on the other hand, because single-parallelism computation can be performed directly without further caching or distribution operations, local memory copying is also unnecessary, thus reducing memory resource consumption.

[0011] As can be seen, the solution implemented in this application can effectively save resource consumption for data query tasks based on distributed database systems and reduce resource waste. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 This is a schematic diagram of an exemplary distributed database system;

[0014] Figure 2A This is a flowchart illustrating the steps of a data processing method according to an embodiment of this application.

[0015] Figure 2B for Figure 2A A schematic diagram illustrating the interaction of task execution operators for communication in a distributed database system according to the embodiment shown.

[0016] Figure 2C for Figure 2A A schematic diagram illustrating a specific process example of the embodiment shown;

[0017] Figure 2D for Figure 2A A schematic diagram illustrating a scenario example of the embodiment shown;

[0018] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0020] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0021] Figure 1 An exemplary system of a distributed database system is shown. For example... Figure 1As shown, the system 100 includes multiple node devices, i.e., data nodes, distributed in different physical locations. These data nodes can be interconnected via network 102, or some data nodes can be grouped into a local database system and then interconnected via network 102. In practical applications, depending on the specific needs, the distributed database system can take different forms, and various other devices and nodes can be added as needed, such as load balancing devices, database management devices, etc.

[0022] Based on a distributed database system, this application provides a data processing method, which will be described below through several embodiments.

[0023] Reference Figure 2A The diagram illustrates a flowchart of the steps of a data processing method according to an embodiment of this application.

[0024] The data processing method in this embodiment includes the following steps:

[0025] Step S202: Based on the parsing results of the data query statement, determine the multiple computation stages of the corresponding data query task in the distributed database system.

[0026] The data query statement can be any statement that a distributed database system can accept and recognize, such as SQL (Structured Query Language) statements. For users, they can input a query request in natural language. The request recipient, such as a browser or search engine, will parse and convert this request into the data query statement described in this application embodiment, and then hand it over to the cloud or server for processing. Of course, this part of the work can also be implemented by the backend through corresponding functional parts, all of which are within the protection scope of this application.

[0027] Parsing a data query statement means resolving it into a logical plan that the database can execute. Based on this logical plan, the distributed database management system (DBMS) can retrieve data from the corresponding nodes, then distribute the retrieved data to different nodes for processing until the result data is obtained, and finally return it to the client through the corresponding nodes. Since the execution of the logical plan may involve multiple computational stages, and the data and nodes involved in different computational stages may also differ, different task divisions are assigned to each computational stage to facilitate the smooth and effective execution of the logical plan. Each computational stage handles different tasks.

[0028] Based on this, in one feasible approach, the data query statement can be parsed, and an abstract syntax tree (AST) can be constructed to construct the data query task corresponding to the query statement. Semantic transformation is then performed on the AST to obtain the corresponding logical plan nodes for executing the data query task. Based on the logical plan nodes and according to the data source sharding rules of the distributed database system, the data query task is divided into multiple computational stages. Using the AST, the logical plan corresponding to the query task can be determined quickly and accurately. The data source sharding rules indicate the storage rules of the source data in the distributed database system. These rules allow for the rapid determination of the node locations where the data required by the data query task resides, enabling access to the source data stored in the corresponding nodes.

[0029] It should be noted that some database query statements have row count limits. These row count limits can be converted into nodes of the corresponding row count limit class in the abstract syntax tree for subsequent data processing. For example, in the SQL statement "Select class,count()from students group by class limit 10", "limit 10" limits the number of rows in the result, which is the row count limit condition. However, those skilled in the art should understand that the above is merely an illustrative example; in data query statements using SQL as an example, row count limits follow the rules of SQL.

[0030] For example, a syntax analysis tool for data query statements can be used to perform syntax analysis on data query statements such as SQL statements to obtain an AST (Abstract Syntax Tree), and then it can be converted into different logical plan nodes according to different semantics.

[0031] During the conversion process, row count constraints can be converted into corresponding row count constraint types of nodes. For example, there are simple row count constraints like `LimitNode` (e.g., constraints imposed by "limit" in SQL statements), ordered row count constraints like `TopNNode` (e.g., constraints imposed by methods such as "row_number()" or "top()" in SQL statements), and row count constraints with special deduplication features like `DistinctLimitNode` (e.g., constraints imposed by "distinct" and "limit" in SQL statements). Subsequently, row count constraint operators can be constructed based on these three types of nodes. Furthermore, these row count constraint operators can immediately stop computation if the row count constraint is met, without needing to receive all input data. This allows network communication to be disconnected promptly, reducing hardware resource waste.

[0032] After obtaining the abstract syntax tree and its corresponding logical plan nodes, multiple computation stages for the corresponding data query task can be obtained according to the data source sharding rules. In one feasible approach, these multiple computation stages may include: a data source stage for retrieving the source data required for the data query statement from multiple distributed data nodes storing source data in the distributed database system; a general stage for data interaction with distributed data nodes in the upstream and downstream computation stages; and an output stage for data interaction with distributed data nodes in the upstream computation stage and the client. This approach fully utilizes the hardware resources of the distributed database system, ensuring smooth task execution. Furthermore, it can improve the parallelism of task execution.

[0033] For example, in order to make full use of the hardware resources of a distributed database system, a local computation phase can be performed on a node close to the data source shard according to the data source sharding rules, and the result of the local computation phase can be used as the input of the global computation phase. This avoids processing all data on the same node and maximizes parallelism while ensuring the correctness of data processing.

[0034] These computational stages can be further categorized into three types based on their data input and output characteristics: SourceStage, CommonStage, and OutputStage. SourceStage retrieves the data required for the data query task from nodes storing source data. CommonStage interacts with other computational stages, including its upstream SourceStage and downstream OutputStage, receiving data as input and outputting data to other stages like OutputStage. OutputStage, in addition to receiving data, outputs the results to the client, thus interacting with both upstream computational stages and the client. However, this is not the only possible approach. Those skilled in the art can further subdivide the computational stages according to actual needs, as described in this example, to facilitate data processing; such subdivisions are also within the scope of this application.

[0035] In addition, unless otherwise specified, in the embodiments of this application, "multiple", "various" and other quantities related to "multiple" all mean two or more.

[0036] Step S204: Based on the computation tasks and parallelism of the computation tasks corresponding to the multiple computation stages, create corresponding task execution operators for each of the multiple computation stages.

[0037] Specifically, in the computation phase of a multi-parallelism computation task, a task execution operator based on inter-process communication is used; in the computation phase of a single-parallelism computation task, a task execution operator based on thread communication is used.

[0038] Different computational stages correspond to different computational tasks. For example, SourceStage primarily acquires the source data required for data query tasks and then hands it over to the computation plan of the CommonStage for processing. Therefore, its corresponding computational tasks include reading source data, data aggregation of the data nodes involved in this stage, and outputting the aggregated data to the data nodes of CommonStage. Analyzing it from the perspective of inter-node communication, it is evident that the computational tasks in this stage typically involve multiple data nodes, and the efficiency of executing tasks in parallel is relatively high. Therefore, it can be considered a computational stage involving multiple degrees of parallelism.

[0039] For example, CommonStage needs to pull data from multiple data nodes of SourceStage for processing, and then output the processing results to OutputStage. Although CommonStage may involve fewer data nodes than SourceStage, it usually still involves multiple data nodes. Moreover, compared to the local data aggregation operations of the data nodes in SourceStage, the aggregation operations of the data nodes involved in CommonStage can be considered as more global data aggregation operations because they need to process and aggregate data from multiple data nodes. From the perspective of inter-node communication, it can be seen that the computation task in this stage also involves multiple data nodes. Executing the task in parallel is highly efficient, and it can be considered as a multi-parallel computation task and a computation stage for multi-parallel tasks.

[0040] For example, the OutputStage is typically a data node that interacts with the client, usually a single node. It pulls data from multiple data nodes involved in the CommonStage, processes it, and outputs it to the client. Analyzing it from the perspective of inter-node communication, it's clear that the number of data nodes involved in this stage's computation is significantly reduced compared to other computation stages, while the primary target for output is the client. Although tasks can be executed in parallel, the resulting resource burden would be greater than with non-parallel methods. Therefore, a single-parallelism approach, i.e., a serial approach, can be adopted. From the perspective of inter-node communication, this can be considered a single-parallelism computation task and a single-parallelism task computation stage.

[0041] Based on the above analysis, corresponding task execution operators can be created for each of the multiple computation stages. This application focuses primarily on task execution operators used for inter-node communication. In one feasible approach, based on the computational tasks corresponding to multiple computation stages, the computation stage with data input and data output tasks can be determined from among the multiple computation stages; a process communication-based task execution operator can be created for the determined computation stage; the parallelism of the computational tasks corresponding to the determined computation stage can be determined; if it is a single-parallelism computational task, then a thread-based task execution operator is used to replace the already created process communication-based task execution operator. In this approach, initially, the parallelism of computational tasks is not distinguished, and process communication-based task execution operators are created for all tasks to achieve communication between data nodes. Furthermore, if a single-parallelism computational task is determined, then a thread-based task execution operator is used to replace the process communication-based task execution operator. In this way, since single-parallelism computational tasks are relatively few in the total computational tasks involved in the entire data query task, it is not necessary to distinguish them each time an operator is built, which can greatly improve the construction efficiency of operators used for communication. Subsequently, a unified judgment can be made, or the judgment can be made when the task is about to be executed, which can effectively improve the overall efficiency of operator construction and execution. However, it is not limited to this. In practical applications, the parallelism of the computation task can also be judged when creating the task execution operator. If the judgment result is single parallelism, then the task execution operator based on thread communication can be directly created without replacing it. This approach is also applicable to the solution in the embodiments of this application.

[0042] In one feasible approach, the task execution operator based on inter-process communication can include: a remote exchange operator for retrieving output data from a remote process via a process, and a local exchange operator for distributing the data retrieved by the remote exchange operator to the local execution thread via a process; the task execution operator based on thread communication includes: an exchange client operator for retrieving output data from a remote process via a thread. By configuring the task execution operators in this way, the actual task requirements can be effectively met, while avoiding resource waste caused by excessive configuration.

[0043] With the computation phases divided, the construction of task execution operators for each computation phase can be achieved as follows: based on the computation tasks and parallelism of the data source phase, create multi-parallelism task output operators for the data source phase; based on the computation tasks and parallelism of the ordinary phase, create multi-parallelism task output operators, multi-parallelism remote exchange operators, and multi-parallelism local exchange operators for the ordinary phase; based on the computation tasks and parallelism of the output phase, create a single-parallelism exchange client operator for the output phase.

[0044] Among them, the task output operator is used to output data to other data nodes, such as data nodes in the downstream computing stage, for example, outputting data through a remote process; the remote exchange operator is used to pull data, such as pulling output data from a remote process; the local exchange operator is used to distribute received data to local threads, for example, distributing output data from a remote process pulled by the remote exchange operator to the local execution thread; and the exchange client operator is used to directly pull output data from a remote process through a thread.

[0045] For example, combining the aforementioned SourceStage, CommonStage, and OutputStage, since SourceStage itself can obtain the source data required for the computation task from the data nodes storing source data to provide input data to the data nodes of other computation stages, from a communication perspective, a TaskOutput operator (task output operator) based on inter-process communication needs to be created for it to output data; CommonStage has the function of interacting with other computation stages for data input and output, so from a communication perspective, TaskOutput operator, RemoteExchange operator (remote exchange operator based on inter-process communication), and LocalExchange operator (local exchange operator based on inter-process communication) can be created for it simultaneously. In addition to the function of receiving data, OutputStage also needs to output the result data to the client, so TaskOutput operator, RemoteExchange operator, and LocalExchange operator are also created for it simultaneously.

[0046] However, to optimize the functionality of the task execution operators used for communication, they can be optimized based on the parallelism of the computation tasks. In one example, the need to remove the RemoteExchange and LocalExchange operators is determined based on the parallelism of the computation tasks, such as the parallelism of computation tasks in certain global computation stages (e.g., OutputStage). To improve computational performance, multiple TaskDrivers that can be executed in parallel can be created in the process for a single computation stage. The parallelism of the TaskDriver is determined by the task execution operators used for communication contained in the computation stage. Generally, the parallelism of the task execution operators used for communication in computation stages without grouping functionality is 1, while the parallelism of the task execution operators used for communication in computation stages with grouping functionality is a fixed value N (N is greater than or equal to 2). If the parallelism is 1, the RemoteExchange and LocalExchange operators are removed, and the ExchangeClient operator (exchange client operator) is directly nested in the source operator to read the input data. That is, the data output by the remote process of CommonStage is directly used as the input of the task execution operator of the next functional class through the ExchangeClient operator, thus saving unnecessary memory copying.

[0047] Furthermore, it should be noted that the number of network links for the TaskOutput and RemoteExchange operators is the product of the number of interactive processes in the upstream and downstream computation stages, and is independent of the number of TaskDrivers. Therefore, these two types of operators greatly improve the scalability of the distributed database system.

[0048] Furthermore, the task output operator and the operator used for data computation, as well as the remote exchange operator and the operator used for data computation, interact asynchronously using a shared blocking queue. From a classification perspective, both task output operators and remote exchange operators involve data interaction between data nodes and can be called interaction operators. Within a data node, there are also operators that perform specific functions, such as data filtering and sorting, which can be called function operators. Therefore, interaction operators and function operators can interact asynchronously using a shared blocking queue to ensure that function operators can obtain data without obstacles, avoid data waiting or congestion, and improve the execution efficiency of data query tasks.

[0049] Step S206: Use the task execution operators corresponding to multiple computation stages to execute the data query task.

[0050] Once the computational stages and their corresponding task execution operators are determined, the data query task can be executed according to the execution logic (obtainable through the logical plan node). Data interaction between data nodes in different computational stages can be performed using task execution operators for communication, such as task execution operators based on inter-process communication or task execution operators based on thread communication.

[0051] In practical implementation, if the computation stage corresponds to a multi-parallel computation task, a remote exchange operator can be used to retrieve data output from the remote process in the upstream computation stage, and then a local exchange operator distributes the retrieved data to multiple execution threads locally. If the computation stage corresponds to a single-parallel computation task, a exchange client operator is used to directly retrieve data output from the remote process in the upstream computation stage and perform data computation processing. This enables efficient data interaction between different data nodes, reduces the number of network communication links through inter-process interaction, and optimizes high-frequency data interactions, such as data interactions generated by computation tasks in the OutputStage computation stage, by using threads to avoid unnecessary local memory copies.

[0052] Furthermore, when using task execution operators corresponding to multiple computation stages, if it is determined that the task execution operator to be used is a row-number-limited operator, then the data query task is executed according to the limiting conditions corresponding to the row-number-limited operator. This allows for timely disconnection of network communication between data nodes, reducing waste of hardware resources.

[0053] The following, combined with Figure 2B This paper provides an exemplary description of the process by which data nodes interact with each other via network communication during different computational stages of the aforementioned data query task.

[0054] Figure 2B In the diagram, from bottom to top, the data source stage, the normal stage, and the output stage are arranged in sequence. Each box represents a data node. Within a data node, the solid horizontal boxes at the bottom represent the RemoteExchange and LocalExchange operators, the hollow horizontal boxes at the top represent the TaskOutput operator based on the buffer, and the hollow vertical boxes in the middle represent the threads that execute the computation tasks of this data node (the number of threads in each data node in the diagram is only an example; those skilled in the art should understand that it can be set according to actual needs in actual use).

[0055] The SourceStage computation task involves data nodes 1, 2, and 3, which store source data. After collecting the source data required for the data query task, data nodes 1, 2, and 3 perform their own data aggregation (partial aggregation operation) and store the aggregation results in their respective buffers, which are then output by the TaskOutput operator.

[0056] In this example, the CommonStage data nodes include data nodes 4 and 5. After data nodes 1, 2, and 3 output data, data nodes 4 and 5 retrieve the data output by these data nodes using their own RemoteExchange operators. As shown in the diagram, data node 4 retrieves data from the first cache of data nodes 1, 2, and 3 using its own RemoteExchange operator; data node 5 retrieves data from the second cache of data nodes 1, 2, and 3 using its own RemoteExchange operator. Then, data node 4 distributes the retrieved data to its multiple execution threads using its own LocalExchange operator. These execution threads can be threads that perform specific functions, such as data filtering, sorting, data aggregation, and other specific computational tasks. After these execution threads complete their respective computational tasks, the corresponding computation results are stored in data node 4's own cache and output by data node 4's own TaskOutput operator. Similarly, data node 5 distributes the pulled data to its multiple execution threads through its own LocalExchange operator. After these multiple execution threads complete their respective computation tasks, the corresponding computation results are stored in data node 5's own cache and output by data node 5's own TaskOutput operator.

[0057] In OutputStage, the corresponding functions are implemented through data node 6, requiring no additional data nodes. Furthermore, this stage involves global computation tasks, requiring no grouping, and its parallelism is 1, i.e., single parallelism. Based on this, data node 6 no longer contains the RemoteExchange and LocalExchange operators; instead, the ExchangeClient operator, through a thread, directly reads the data output from data nodes 4 and 5, eliminating the need for memory copying. Subsequently, after performing the corresponding global computation on the read data, data node 6 stores it in its own cache and outputs it through its own TaskOutput operator for interaction with the client.

[0058] As shown in the diagram, data node 6 reduced the number of network communication links by removing the RemoteExchange and LocalExchange operators, and avoided local memory copying by using the ExchangeClient operator. This saved on hardware resource consumption.

[0059] In this embodiment, when querying data based on a distributed database system, different task execution operators are constructed for different computational tasks and their parallelism at different computational stages. Data query tasks are then executed based on these operators. Specifically, if a computational stage requires multi-parallelism computational tasks, a task execution operator based on inter-process communication (IPC) is used. This type of operator uses IPC to achieve data interaction between upstream and downstream computational tasks, such as data retrieval, caching, and distribution. Compared to the traditional method of achieving data interaction between upstream and downstream computational tasks through inter-thread communication, this significantly reduces the number of network communication links, thereby reducing hardware resource consumption. Conversely, if a computational stage requires single-parallelism computational tasks, a task execution operator based on thread communication is used. This type of operator can directly retrieve data for corresponding computation without caching or distribution operations. On the one hand, in the case of single-parallelism, it ensures the smooth execution of the computational task; on the other hand, because single-parallelism computation can be performed directly without caching or distribution operations, local memory copying is also unnecessary, thus reducing memory resource consumption.

[0060] As can be seen, the solution in this embodiment can effectively save resource consumption for data query tasks based on distributed database systems and reduce resource waste.

[0061] The following is based on the above. Figure 2B Based on the data interaction process, the process of implementing a data query task is illustrated by example, such as... Figure 2C As shown. Furthermore, in this example, the data query statement is an SQL statement.

[0062] As shown in the figure, the process includes:

[0063] Step 1: Parse the SQL statement to obtain the corresponding AST.

[0064] For example, by using an appropriate SQL syntax parsing tool, the SQL statement can be parsed to obtain an AST.

[0065] Step 2 involves transforming the AST into different logical plan nodes based on different semantics. This includes: transforming simple row count constraints into LimitNodes, ordered row count constraints into TopNNodes, and row count constraints with deduplication features into DistinctLimitNodes.

[0066] As mentioned earlier, the three types of nodes involving row count limitations are all row count-limited nodes, and corresponding row count-limited operators can be constructed based on these nodes. These row count-limited operators can stop immediately during computation if the row count limitation is met, without needing to receive all input data.

[0067] Process 3: Determine the corresponding logical plan based on the logical plan nodes, and break down the logical plan into multiple calculation stages.

[0068] To fully utilize the hardware resources of a distributed database system, the logical plan's local computation phase can be executed first on data nodes closest to the data source shard, according to the data source sharding rules. The results of these local computation phases can then be used as input for the global computation phase. This avoids processing all data on the same data node, maximizing parallelism while ensuring correctness.

[0069] These computation stages are further divided into three categories based on their data input and output characteristics: SourceStage, CommonStage, and OutputStage. SourceStage itself provides input data, and TaskOutput operators can be created for it to facilitate interaction between data nodes. CommonStage has the function of interacting with other computation stages for data input and output, and TaskOutput, RemoteExchange, and LocalExchange operators are created for it. OutputStage, in addition to receiving data, also outputs the data results obtained from data query tasks to the client, and therefore TaskOutput, RemoteExchange, and LocalExchange operators are also created for it. It is evident that CommonStage and OutputStage share the same characteristics, while SourceStage differs from both.

[0070] Step 4: Determine if the current computation stage is SourceStage; if yes, proceed to Step 5; otherwise, proceed to Step 6.

[0071] Because SourceStage has different characteristics from CommonStage and OutputStage, it needs to be handled separately.

[0072] Step 5: Create the TaskOutput operator. Then, proceed to step 7 for execution.

[0073] In step 6, the TaskOutput, RemoteExchange, and LocalExchange operators are created simultaneously. Then, proceed to step 7 for execution.

[0074] Step 7: Determine if the parallelism of the computation task in the current computation stage is 1; if yes, proceed to step 8; otherwise, proceed to step 9.

[0075] Among them, whether the parallelism is 1 or not is also known as whether it is single parallelism.

[0076] In step 8, remove the RemoteExchange and LocalExchange operators and replace them with the ExchangeClient operator. Then, proceed to step 9.

[0077] To improve computational performance, multiple TaskDrivers are created within a process for each computation stage, allowing for parallel execution. The parallelism of a TaskDriver is determined by the characteristics of the task execution operators within the current computation stage. Computation stages without grouping capabilities have a parallelism of 1, while those with grouping capabilities have a fixed parallelism of N (N greater than or equal to 2). If the parallelism is 1, the RemoteExchange and LocalExchange operators are removed, and the ExchangeClient operator is nested directly within the source operator to read input data, thus avoiding unnecessary memory copying.

[0078] In process 9, if the operator to be executed is determined to be a row data-limited operator LimitNode, TopNNode, or DistinctLimitNode, then fast short-circuiting is performed when the row number limit condition is met.

[0079] In this context, "fast short circuit" refers to the timely disconnection of network communication to avoid wasting hardware resources.

[0080] For operators with row count limitations such as LimitNode, TopNNode, and DistinctLimitNode, if the row count limitation condition is met, these operators can be quickly short-circuited. At the same time, an end request is sent to the TaskOutput operator on the data provider side to clear the corresponding memory buffer, without transmitting invalid input data.

[0081] After all logical plans have been executed, the result data of the data query task will be obtained. For example, this can be obtained from the data nodes of OutputStage. Figure 2B Data node 6, as shown, sends the result data to the client.

[0082] As can be seen from this example, inter-process communication between different data nodes is first collected through the RemoteExchange operator and then distributed through the LocalExchange operator. This results in a small number of network connections, which is the product of the number of communicating processes, making the distributed system highly scalable. Furthermore, for the computation phase with a parallelism of 1, there is no need to create RemoteExchange and LocalExchange operators, resulting in fast execution speed and avoiding unnecessary memory copying. In addition, for row-limited operators that can be terminated early, short-circuit information is fed back to the data provider, and the memory buffer corresponding to invalid TaskOutput operators is cleared in a timely manner, improving memory utilization.

[0083] The following example demonstrates how to query data from stored cloud logs. Figure 2D The specific application scenarios of the above process are illustrated by example.

[0084] The process of querying data from cloud logs includes: receiving a data query statement for querying stored cloud logs; logically parsing the data query statement and determining the multiple computation stages of the corresponding data query task in the distributed database system storing cloud logs based on the parsing results; creating corresponding task execution operators for each computation stage based on the computation tasks and their parallelism; wherein, in the computation stages corresponding to multi-parallelism computation tasks, task execution operators based on inter-process communication are used; in the computation stages corresponding to single-parallelism computation tasks, task execution operators based on thread communication are used; and the data query task for the cloud logs is executed using the task execution operators corresponding to the multiple computation stages.

[0085] In one feasible approach, creating corresponding task execution operators for multiple computation stages based on the computation tasks and parallelism of the computation tasks can be achieved as follows: Based on the computation tasks corresponding to the multiple computation stages, determine the computation stage with data input and data output tasks from among the multiple computation stages; create a task execution operator based on inter-process communication for the determined computation stage; determine the parallelism of the computation tasks corresponding to the determined computation stage; if it is a single-parallelism computation task, replace the already created task execution operator based on inter-process communication with a task execution operator based on thread communication.

[0086] Optionally, the task execution operator based on process communication includes: a remote exchange operator for pulling remote process output data through a process, and a local exchange operator for distributing the data pulled by the remote exchange operator to the local execution thread through a process; the task execution operator based on thread communication includes: an exchange client operator for pulling remote process output data through a thread.

[0087] In one feasible approach, task execution operators corresponding to multiple computation stages are used to execute data query tasks. This can include: if the computation stage corresponds to a multi-parallel computation task, a remote exchange operator is used to pull data output by a remote process in the upstream computation stage, and the pulled data is distributed to multiple execution threads locally through a local exchange operator; if the computation stage corresponds to a single-parallel computation task, an exchange client operator is used to directly pull data output by a remote process in the upstream computation stage through a thread, and perform data computation processing.

[0088] In one feasible approach, a data query task is executed using task execution operators corresponding to multiple computation stages, including: if it is determined that the task execution operator to be used is a row number-limited operator, then the data query task is executed according to the limiting conditions corresponding to the row number-limited operator.

[0089] Optionally, the multiple computation stages include: a data source stage for obtaining the source data required for the data query task from multiple distributed data nodes storing source data in the distributed database system; a general stage for data interaction with distributed data nodes in the upstream computation stage and distributed data nodes in the downstream computation stage; and an output stage for data interaction with distributed data nodes in the upstream computation stage and the client.

[0090] In one feasible approach, based on the computational tasks and their parallelism corresponding to multiple computational stages, corresponding task execution operators are created for each stage, including: creating a multi-parallelism task output operator for the data source stage based on the computational tasks and their parallelism corresponding to the data source stage; creating a multi-parallelism task output operator, a multi-parallelism remote exchange operator, and a multi-parallelism local exchange operator for the normal stage based on the computational tasks and their parallelism corresponding to the normal stage; and creating a single-parallelism exchange client operator for the output stage based on the computational tasks and their parallelism corresponding to the output stage.

[0091] Optionally, the task output operator and the operator used for data computation, as well as the remote exchange operator and the operator used for data computation, interact asynchronously using a shared blocking queue.

[0092] In one feasible approach, based on the parsing results of the data query statement, the corresponding data query task is determined to have multiple computational stages in the distributed database system. This includes: performing syntactic analysis on the data query statement and constructing an abstract syntax tree for executing the data query task corresponding to the data query statement based on the analysis results; performing semantic transformation on the abstract syntax tree to obtain the corresponding logical plan node for executing the data query task; and, based on the logical plan node, splitting the data query task into multiple computational stages according to the data source sharding rules of the distributed database system.

[0093] For example, such as Figure 2D As shown, suppose a user enters a query request through the client interface stating "Query logs of data updates performed on the XX cloud platform on January 1, 2023". This query request will first be converted into a corresponding SQL statement. In this example, it is assumed that the statement is converted into an SQL statement on the client side. Then, the request information carrying the SQL statement will be sent to the cloud. After receiving the request and obtaining the SQL statement, the cloud parses it and generates an Abstract Syntax Tree (AST) based on the parsing result. Then, based on this AST, the corresponding logical plan nodes and the corresponding logical plan are obtained. According to the data source sharding rules, the logical plan is split into three computation stages to implement the logical plan: SourceStage, CommonStage, and OutputStage. Then, corresponding task execution operators are constructed for each of these three computation stages, including task execution operators for communication. This example mainly illustrates the task execution operators for communication. For reference, see... Figure 2C As shown, TaskOutput operators can be constructed for SourceStage, TaskOutput, RemoteExchange, and LocalExchange operators can be constructed for CommonStage, and ExchangeClient operators can be constructed for OutputStage.

[0094] Based on this, such as Figure 2B As shown, the corresponding computational tasks are executed sequentially through the task execution operators of the functional classes in each computational stage. After the computational tasks are completed, communication and data interaction between the data nodes in each computational stage are achieved through the task execution operators of the interaction classes used for communication in each computational stage, as described above. This process continues until all computational tasks involved in the data query task are completed, at which point the result data is obtained.

[0095] After obtaining the result data, it will be sent to the client by the data nodes of OutputStage. Upon receiving the result data, the client will display it on the interface according to certain rules. For example, if the client has a corresponding display template set up, the result data can be combined with that template and displayed to the user, and so on.

[0096] This embodiment enables fast and efficient data access to cloud logs stored in the cloud, and saves resources for data query tasks based on distributed database systems, thus reducing resource waste.

[0097] It should be noted that the specific implementation of each step in this embodiment is relatively simple, and relevant parts can be referred to the corresponding parts of the previous embodiments.

[0098] Reference Figure 3 The diagram shows a structural schematic of an electronic device according to Embodiment 5 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0099] like Figure 3 As shown, the electronic device may include: a processor 302, a communications interface 304, a memory 306, and a communications bus 308.

[0100] in:

[0101] The processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308.

[0102] Communication interface 304 is used to communicate with other electronic devices or servers.

[0103] The processor 302 is used to execute program 310, specifically to perform the relevant steps in the above data processing method embodiment.

[0104] Specifically, program 310 may include program code that includes computer operation instructions.

[0105] Processor 302 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0106] Memory 306 is used to store program 310. Memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0107] Program 310 may include multiple computer instructions. Specifically, program 310 may use multiple computer instructions to cause processor 302 to perform the operation corresponding to the data processing method described in any of the foregoing multiple method embodiments.

[0108] The specific implementation of each step in procedure 310 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0109] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0110] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the data processing methods in the above-described multiple method embodiments.

[0111] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, user input data, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, data used for querying, data displayed, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0112] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0113] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0114] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0115] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A data processing method, comprising: Based on the parsing results of the data query statement, determine the multiple computation stages of the corresponding data query task in the distributed database system; Based on the computation tasks corresponding to the multiple computation stages and the parallelism of the computation tasks, corresponding task execution operators are created for each of the multiple computation stages; wherein, in the computation stages corresponding to multi-parallelism computation tasks, task execution operators based on inter-process communication are used; and in the computation stages corresponding to single-parallelism computation tasks, task execution operators based on thread communication are used. The data query task is executed using the task execution operators corresponding to the multiple computation stages.

2. The method according to claim 1, wherein, The step of creating corresponding task execution operators for each of the multiple computation stages based on the computation tasks corresponding to the multiple computation stages and the parallelism of the computation tasks includes: Based on the computation tasks corresponding to the plurality of computation stages, determine the computation stage with data input tasks and data output tasks from the plurality of computation stages; To define the computation phase, a task execution operator based on inter-process communication is created; Determine the degree of parallelism of the computation task corresponding to the determined computation stage; If it is a single-parallel computation task, then the previously created task execution operator based on process communication is replaced with a task execution operator based on thread communication.

3. The method according to claim 2, wherein, The task execution operator based on inter-process communication includes: a remote exchange operator for pulling output data from a remote process through a process, and a local exchange operator for distributing the data pulled by the remote exchange operator to the local execution thread through a process; The task execution operator based on thread communication includes: an exchange client operator for pulling remote process output data via threads.

4. The method according to claim 3, wherein, The step of using the task execution operators corresponding to the multiple computation stages to execute the data query task includes: If the computation phase corresponds to a computation task with multiple degrees of parallelism, then the remote exchange operator is used to pull data output by the remote process in the upstream computation phase, and the local exchange operator is used to distribute the pulled data to multiple execution threads locally. If the computation phase corresponds to a single-parallel computation task, then the aforementioned exchange client operator is used to directly pull the data output by the remote process in the upstream computation phase through a thread, and perform data computation processing.

5. The method according to any one of claims 1-4, wherein, The step of using the task execution operators corresponding to the multiple computation stages to execute the data query task includes: During the process of using the task execution operators corresponding to the multiple calculation stages, if it is determined that the task execution operator to be used is a row number-limited operator, then the data query task is executed according to the limiting conditions corresponding to the row number-limited operator.

6. The method according to any one of claims 1-4, wherein, The multiple computational stages include: The data source stage is used to obtain the source data required for the data query task from multiple distributed data nodes storing source data in the distributed database system. A normal stage used for data interaction with distributed data nodes in the upstream computing stage and distributed data nodes in the downstream computing stage. and, The output stage is used for data interaction with distributed data nodes and clients in the upstream computing stage.

7. The method according to claim 6, wherein, The step of creating corresponding task execution operators for each of the multiple computation stages based on the computation tasks corresponding to the multiple computation stages and the parallelism of the computation tasks includes: Based on the computation task corresponding to the data source stage and the parallelism of the computation task, create a multi-parallelism task output operator for the data source stage; Based on the computational task corresponding to the normal stage and the parallelism of the computational task, create a multi-parallelism task output operator, a multi-parallelism remote exchange operator, and a multi-parallelism local exchange operator for the normal stage. Based on the computation task corresponding to the output stage and the parallelism of the computation task, a single-parallelism exchange client operator is created for the output stage.

8. The method according to claim 7, wherein, The task output operator and the operator used for data computation, as well as the remote exchange operator and the operator used for data computation, interact asynchronously using a shared blocking queue.

9. The method according to any one of claims 1-4, wherein, The step of determining the multiple computational stages of the corresponding data query task in the distributed database system based on the parsing results of the data query statement includes: The data query statement is subjected to syntax analysis, and an abstract syntax tree is constructed based on the analysis results to execute the data query task corresponding to the data query statement. The abstract syntax tree is semantically transformed to obtain the corresponding logical plan node for executing the data query task; Based on the logical plan node, and in accordance with the data source sharding rules of the distributed database system, the data query task is divided into multiple computation stages.

10. A data processing method, comprising: Receive data query statements used to query data from stored cloud logs; The data query statement is logically parsed, and the corresponding data query task is determined in multiple computation stages in the distributed database system storing the cloud logs based on the parsing results. Based on the computation tasks corresponding to the multiple computation stages and the parallelism of the computation tasks, corresponding task execution operators are created for each of the multiple computation stages; wherein, in the computation stages corresponding to multi-parallelism computation tasks, task execution operators based on inter-process communication are used; and in the computation stages corresponding to single-parallelism computation tasks, task execution operators based on thread communication are used. The task execution operators corresponding to the multiple computing stages are used to execute data query tasks for the cloud logs.

11. The method according to claim 10, wherein, The step of creating corresponding task execution operators for each of the multiple computation stages based on the computation tasks corresponding to the multiple computation stages and the parallelism of the computation tasks includes: Based on the computation tasks corresponding to the plurality of computation stages, determine the computation stage with data input tasks and data output tasks from the plurality of computation stages; To define the computation phase, a task execution operator based on inter-process communication is created; Determine the degree of parallelism of the computation task corresponding to the determined computation stage; If it is a single-parallel computation task, then the previously created task execution operator based on process communication is replaced with a task execution operator based on thread communication.

12. The method according to claim 11, wherein, The task execution operator based on inter-process communication includes: a remote exchange operator for pulling output data from a remote process through a process, and a local exchange operator for distributing the data pulled by the remote exchange operator to the local execution thread through a process; The task execution operator based on thread communication includes: an exchange client operator for pulling remote process output data via threads.

13. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-12.

14. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-12.

Images