Task execution method and device
By optimizing operator combinations in big data scenarios and forming a target operator combination to replace the first operator graph, the bottleneck of task execution efficiency improvement is solved, and more efficient task execution is achieved.
Patent Information
- Application Number
- CN202310858927.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-12
- Publication Date
- 2025-01-14
AI Technical Summary
Existing technologies have limited room for improvement in task execution efficiency in big data scenarios, especially in terms of operator independence and composability.
By determining the first operator graph corresponding to the task to be processed, matching the target operator combination, and replacing parts of the first operator graph to form a second operator graph, which includes the target operator combination, the data caching requirements are reduced, and the task execution performance and efficiency are improved.
It improves the performance and efficiency of task execution, especially in big data scenarios, reduces the demand for caching resources, and is suitable for a wider range of application scenarios.
Smart Images
Figure CN119311397A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a task execution method, a task execution device, an electronic device, and a computer-readable storage medium. Background Art
[0002] With the rapid development of computer technology and Internet technology, the efficiency of computer tasks has important implications for scenarios such as big data.
[0003] The task execution in the existing technology is usually performed by operators that are divided into more fine-grained parts. The independence and composability of operators make it possible to improve the efficiency of task execution by executing operators and then executing tasks. However, the efficiency of task execution in this task execution method still has a lot of room for improvement in scenarios such as big data. Summary of the invention
[0004] The present application provides a task execution method and device, which can improve the task execution efficiency by optimizing operators.
[0005] In a first aspect, the present application relates to a task execution method, comprising: determining a target operator combination that at least partially matches the first operator graph according to a first operator graph corresponding to a task to be processed. Determine a second operator graph corresponding to the task to be processed according to the target operator combination and the first operator graph. Execute the second operator graph based on received task data to be processed to obtain an execution result of the task to be processed. The target operator combination includes multiple operators.
[0006] An operator graph can be understood as a set of operators represented in the form of a graph, where each node of the operator graph represents an independent operator.
[0007] Exemplarily, the first operator graph corresponding to the task to be processed may be obtained by converting the task to be processed into an operator graph after determining the task to be processed, and the first operator graph may represent the operator execution flow of the task to be processed.
[0008] The target operator combination at least partially matches the first operator graph, which means that the target operator combination and the first operator graph are equivalent based on the matching parts. For example, the target operator combination can be used to replace the part of the first operator graph that matches the target operator combination, and a second operator graph can be obtained. The second operator graph can still represent the operator execution process of the task to be processed.
[0009] The difference between the first operator graph and the second operator graph is that all nodes of the first operator graph are independent single operators, while the second operator graph includes a target operator combination (and independent single operators), and the target operator combination as a whole is independent, for example, some nodes of the second operator graph are target operator combinations. Some nodes of the second operator graph are target operator combinations, and for multiple operators included in the target operator combination, data between any two operators does not need to be cached.
[0010] An embodiment of the present application is an operator optimization method, and the operator optimization is reflected in using a target operator combination including multiple operators to completely replace the part of the first operator graph that matches the target operator graph to obtain a second operator graph, and the second operator graph includes the target operator combination. The target operator combination has better performance at least in terms of data caching. Therefore, by executing the second operator graph including the target operator combination, for example, the demand for cache resources for task execution can be reduced, and the performance and efficiency of task execution are higher, which is particularly suitable for scenarios such as big data.
[0011] In addition, some implementations may also optimize operators by, for example, combining "chain calls" between operators into one code. This method is equivalent to executing all operators of the first operator graph as a whole. Although it can also improve the task execution efficiency to a certain extent, integrating all operators of the first operator graph into one code brings additional task execution costs, and this method is limited by the limitations of the integrated operators, and there is little room for optimization of operators. In comparison, the task execution method of the embodiment of the present application is not limited to this, and the task execution performance and efficiency can be improved by optimizing the operators.
[0012] In a possible embodiment, according to a first operator graph corresponding to a task to be processed, determining a target operator combination that at least partially matches the first operator graph includes: taking each node of the first operator graph as a node to be matched, and determining an operator combination including the node to be matched as a candidate operator combination. Matching the candidate operator combination with the first operator subgraph to determine the target operator combination. The first operator subgraph is a subgraph of the first operator graph, and the first operator subgraph includes the node to be matched.
[0013] Exemplarily, for example, each node of the first operator graph may be traversed so that each node of the first operator graph is a node to be matched.
[0014] The task execution method of the embodiment of the present application can perform fuzzy matching based on each operator by taking each node of the first operator graph as a node to be matched and determining the operator combination including the node to be matched as a candidate operator combination. Then, the candidate operator combination is matched with the first operator subgraph to achieve accurate matching of the candidate operator combination with the first operator subgraph, thereby accurately and efficiently determining the target operator combination.
[0015] Exemplarily, the second operator graph may be obtained by replacing the first operator subgraph with a successfully matched target operator combination in the first operator graph.
[0016] In a possible embodiment, matching the candidate operator combination with the first operator subgraph to determine the target operator combination includes: matching the candidate operator combination with the first operator subgraph based on graph isomorphism to determine the target operator combination.
[0017] The edges between nodes can represent the graph structure. If two graphs are isomorphic, then the two graphs can be considered to be equivalent. The task execution method of the embodiment of the present application matches the candidate operator combination with the first operator subgraph based on the isomorphism of the graph, which can adapt to the characteristics of the graph structure and accurately match the candidate operator combination with the first operator subgraph.
[0018] In a possible embodiment, executing the second operator graph based on the received task data to be processed to obtain the execution result of the task to be processed includes: calling an operator combination library to execute a target operator combination based on the received task data to be processed and obtain the execution result of the task to be processed. The operator combination library includes the target operator combination.
[0019] Calling the operator combination library can be understood as calling the interface of the operator combination library, and the interface is the application programming interface.
[0020] The second operator graph is obtained according to the target operator combination and the first operator graph. The second operator graph includes the target operator combination. Therefore, executing the second operator graph includes executing the target operator combination. For example, the target operator combination can be executed by calling an operator combination library.
[0021] In a possible embodiment, when the task to be processed is a subtask of the target task, the method further includes: dividing the target task into a plurality of subtasks, and determining the execution result of the target task according to the execution result of each subtask.
[0022] In some cases, for complex target tasks or for reasons such as execution efficiency, the target task can be divided into multiple subtasks. By executing multiple subtasks and merging the processing results of multiple subtasks, the target task can be executed more efficiently and can also adapt to a wider range of application scenarios.
[0023] In a possible embodiment, the task to be processed or the target task is a query task.
[0024] In a second aspect, the present application relates to a task execution device, comprising: a target operator combination determination module, a second operator graph determination module, and an execution module. The target operator combination determination module is used to determine a target operator combination that at least partially matches the first operator graph based on the first operator graph corresponding to the task to be processed. The target operator combination includes multiple operators. The second operator graph determination module is used to determine a second operator graph corresponding to the task to be processed based on the target operator combination and the first operator graph. The execution module is used to execute the second operator graph based on the received task data to be processed to obtain the execution result of the task to be processed.
[0025] In a third aspect, the present application relates to an electronic device, comprising: at least one processor, at least one processor is used to execute instructions so that the electronic device can perform the above-mentioned task execution method.
[0026] In a fourth aspect, the present application relates to a computer-readable storage medium storing computer instructions, which, when executed, causes the computer to execute the above-mentioned task execution method.
[0027] In a fifth aspect, the present application relates to a computer program product, including a computer program, wherein the computer program is stored on a readable storage medium, and when the computer program is executed, the computer implements the above-mentioned task execution method. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] The following is an introduction to the drawings used in the embodiments of the present application.
[0029] Figure 1 The system architecture of the task execution method and device according to an embodiment of the present disclosure is schematically shown;
[0030] Figure 2 A flowchart of a task execution method according to an embodiment of the present disclosure is schematically shown;
[0031] Figure 3 A schematic diagram schematically shows a task execution method according to another embodiment of the present disclosure;
[0032] Figure 4A A schematic diagram of the first operator graph of stage S is schematically shown;
[0033] Figure 4B A schematic diagram of the second operator graph of stage S is schematically shown;
[0034] Figure 5 A block diagram schematically shows a task execution device according to an embodiment of the present disclosure; and
[0035] Figure 6 A block diagram of a task execution device that can implement an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0037] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0038] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.
[0039] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0040] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0041] The embodiment of the present disclosure provides a task execution method to improve the traditional task execution method.
[0042] Figure 1 The system architecture of the task execution method and device according to an embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0043] like Figure 1 As shown, the system architecture 100 according to this embodiment may include clients 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the clients 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0044] Users can use clients 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on clients 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).
[0045] The clients 101, 102, 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. The clients 101, 102, 103 of the disclosed embodiment may, for example, run applications.
[0046] The server 105 may be a server that provides various services, such as a backend management server that provides support for websites browsed by users using the clients 101, 102, and 103 (for example only). The backend management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the client. In addition, the server 105 may also be a cloud server, that is, the server 105 has a cloud computing function.
[0047] It should be noted that the task execution method provided in the embodiment of the present disclosure can be executed by the server 105. Accordingly, the task execution device provided in the embodiment of the present disclosure can be set in the server 105. The task execution method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the client 101, 102, 103 and / or the server 105. Accordingly, the task execution device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the client 101, 102, 103 and / or the server 105.
[0048] In one example, the server 105 may obtain the pending task data from the clients 101 , 102 , and 103 through the network 104 to execute the pending task or the target task, and obtain the execution result of the pending task or the execution result of the target task.
[0049] It should also be noted that in a distributed scenario, there are multiple servers 105, each of which is a node of the distributed system. The task to be processed or the target task can be divided into multiple subtasks, each of which can be executed by one or more nodes. Then, by integrating the execution results of the corresponding multiple nodes, the execution results of the task to be processed or the target task can be obtained.
[0050] It should be understood that Figure 1 The number of clients, networks and servers in the embodiment is only for illustration. Any number of clients, networks and servers may be provided as required.
[0051] It should be noted that in the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0052] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.
[0053] The present disclosure provides a task execution method. Figure 1 The system architecture of Figure 2 4 to describe the task execution method of the exemplary embodiment of the present disclosure. The task execution method of the exemplary embodiment of the present disclosure may be, for example, Figure 2 The network device 202 shown performs.
[0054] Figure 2 The flowchart of the task execution method 200 according to an embodiment of the present disclosure is schematically shown.
[0055] like Figure 2 As shown, the task execution method 200 according to an embodiment of the present disclosure includes operations S210 to S230.
[0056] In operation S210, according to a first operator graph corresponding to a task to be processed, a target operator combination at least partially matching the first operator graph is determined.
[0057] The target operator combination includes multiple operators. The target operator combination can also be understood as the result of coupling multiple independent single operators.
[0058] Exemplarily, the first operator graph corresponding to the task to be processed may be obtained by converting the task to be processed into an operator graph after determining the task to be processed, and the first operator graph may represent the operator execution flow of the task to be processed.
[0059] An operator graph can be understood as a set of operators represented in the form of a graph, where each node of the operator graph represents an independent operator.
[0060] A graph can be understood as a data structure consisting of a finite non-empty set of vertices and a set of edges between vertices. A graph can be represented as: G(V, E), where G represents a graph, V is the set of vertices in the graph G, and E is the set of edges in the graph G.
[0061] Operator is Operator (abbreviated as OP), which can be understood as operation logic.
[0062] For example, the first operator graph may include X nodes, and these X nodes have a specific association relationship. For example, the target operator combination includes Y nodes, the first operator graph also includes these Y nodes, and the association relationship between the corresponding Y nodes in the first operator graph is the same as the association relationship between the Y nodes of the target operator (X and Y are both integers greater than or equal to 2, and X is greater than or equal to Y), then it can be considered that the target operator combination and the first operator graph at least partially match.
[0063] In operation S220, a second operator graph corresponding to the task to be processed is determined according to the target operator combination and the first operator graph.
[0064] The target operator combination at least partially matches the first operator graph, which means that the target operator combination and the first operator graph are equivalent based on the matching parts. For example, the target operator combination can be used to replace the part of the first operator graph that matches the target operator combination, and a second operator graph can be obtained. The second operator graph can still represent the operator execution process of the task to be processed.
[0065] The difference between the first operator graph and the second operator graph is that all nodes of the first operator graph are independent single operators, while the second operator graph includes a target operator combination (and independent single operators), and the target operator combination as a whole is independent, for example, some nodes of the second operator graph are the target operator combination.
[0066] In operation S230 , the second operator graph is executed based on the received task data to be processed to obtain an execution result of the task to be processed.
[0067] “Executing the second operator graph based on the received task data to be processed” can be understood as executing each node on the received task data to be processed based on the structure of the second operator graph, thereby obtaining the execution result of the task to be processed.
[0068] For example, it may be receiving task data to be processed from a client.
[0069] Each node of the first operator graph is an independent single operator. Taking any two adjacent nodes N1 and N2 as an example, the output data d1 of node N1 needs to be cached. When node N2 uses the output data d1 of node N1 as input data, node N2 needs to obtain the output data d1 of node N1 from the cache space.
[0070] Some nodes of the second operator graph are target operator combinations. For multiple operators included in the target operator combination, data between any two operators does not need to be cached.
[0071] Therefore, according to the task execution method of the embodiment of the present disclosure, it is an operator optimization method. The operator optimization is reflected in using a target operator combination including multiple operators to completely replace the part of the first operator graph that matches the target operator graph to obtain a second operator graph. The second operator graph includes the target operator combination. The target operator combination has better performance at least in terms of data caching. Therefore, by executing the second operator graph including the target operator combination, for example, the demand for cache resources for task execution can be reduced, and the performance and efficiency of task execution are higher, which is particularly suitable for scenarios such as big data.
[0072] In addition, some implementations may also optimize operators by, for example, combining "chain calls" between operators into one code. This method is equivalent to executing all operators of the first operator graph as a whole. Although it can also improve the task execution efficiency to a certain extent, integrating all operators of the first operator graph into one code brings additional task execution costs, and this method is limited by the limitations of the integrated operators, leaving little room for optimization of operators. In comparison, the task execution method of the disclosed embodiment is not limited to this, and the task execution performance and efficiency can be improved by optimizing the operators.
[0073] According to another embodiment of the task execution method of the present disclosure, for example, the following embodiment can be used to implement a specific example of determining a target operator combination that at least partially matches the first operator graph according to the first operator graph corresponding to the task to be processed: taking each node of the first operator graph as a node to be matched, determining the operator combination including the node to be matched as a candidate operator combination. Matching the candidate operator combination with the first operator subgraph to determine the target operator combination.
[0074] The first operator subgraph is a subgraph of the first operator graph, and the first operator subgraph includes nodes to be matched.
[0075] A subgraph can be understood as a graph whose node set and edge set are respectively a subset of the node set and a subset of the edge set of a certain graph. In the task execution method of the embodiment of the present disclosure, the node set of the first operator subgraph is, for example, a subset of the node set of the first operator graph, and the edge set of the first operator subgraph is, for example, a subset of the edge set of the first operator graph. The full first operator subgraph can enumerate all possible combinations of target operators.
[0076] In order to improve the applicability of the target operator combination while achieving operator coupling, the number of operators included in the target operator combination can, for example, take a smaller value. In this case, it is usually difficult for the target operator combination to fully match the first operator graph. The full match here means that the nodes and node associations of the target operator combination are respectively the same as the nodes and node associations of the first operator graph. Usually, the target operator combination can only partially match the first operator graph. The partial match here can be understood as the nodes and node associations of the target operator are respectively the same as the nodes and node associations of the subgraph of the first operator graph, that is, the subgraph of the first operator graph partially matches the target operator combination.
[0077] Of course, in some cases, there will be a target operator combination that completely matches the first operator graph. In this case, the first operator graph can be replaced with the target operator combination, and the obtained second operator graph is the target operator combination.
[0078] Exemplarily, for example, each node of the first operator graph may be traversed so that each node of the first operator graph is a node to be matched.
[0079] According to the task execution method of the embodiment of the present disclosure, by taking each node of the first operator graph as a node to be matched, determining the operator combination including the node to be matched as a candidate operator combination, fuzzy matching can be performed based on each operator. Then, matching the candidate operator combination with the first operator subgraph can achieve accurate matching of the candidate operator combination with the first operator subgraph, thereby accurately and efficiently determining the target operator combination.
[0080] Exemplarily, the second operator graph may be obtained by replacing the first operator subgraph with a successfully matched target operator combination in the first operator graph.
[0081] For example, the operator to be matched can be used as a matching reference, and the operator to be matched can be expanded upward and downward to perform iterations to determine the first operator subgraph, where the upward expansion and downward expansion refer to the operator to be matched as a reference node and expansion to the parent node and child node of the reference node. The first operator subgraph of each iteration is matched with the candidate operator combination.
[0082] According to a task execution method according to another embodiment of the present disclosure, for example, a specific example of matching a candidate operator combination with a first operator subgraph to determine a target operator combination can be implemented using the following embodiment: matching the candidate operator combination with the first operator subgraph based on the isomorphism of the graph to determine the target operator combination.
[0083] The isomorphism of graphs can be understood as follows: assuming G = (V, E) and G1 = (V1, E1) are two graphs, if there exists a bijection m:V→V1 such that for all x, y∈V, xy∈E is equivalent to m(x)m(y)∈E1, then G and G1 are called isomorphic (both G and G1 are graphs).
[0084] The edges between nodes can represent the graph structure. If two graphs are isomorphic, then the two graphs can be considered to be equivalent. According to the task execution method of the embodiment of the present disclosure, the candidate operator combination is matched with the first operator subgraph based on the isomorphism of the graph, which can adapt to the characteristics of the graph structure and accurately match the candidate operator combination with the first operator subgraph.
[0085] According to another embodiment of the task execution method of the present disclosure, for example, the following embodiment can be used to implement a specific example of executing the second operator graph based on the received task data to be processed to obtain the execution result of the task to be processed: calling the operator combination library to execute the target operator combination based on the received task data to be processed and obtain the execution result of the task to be processed.
[0086] The operator combination library includes target operator combinations.
[0087] Calling the operator combination library can be understood as calling the interface of the operator combination library, and the interface is the Application Programming Interface (API for short).
[0088] The second operator graph is obtained according to the target operator combination and the first operator graph. The second operator graph includes the target operator combination. Therefore, executing the second operator graph includes executing the target operator combination. For example, the target operator combination can be executed by calling an operator combination library.
[0089] The second operator graph includes a target operator combination. When the second operator graph also includes an independent single operator that has not been replaced by the target operator combination, for example, the target operator combination and the independent single operator can be executed according to the structure of the second operator graph. For example, the target operator combination can be executed by calling an operator combination library, and the independent single operator can be executed by calling a single operator library, thereby obtaining the execution result of the task to be processed.
[0090] Calling the operator combination library and / or the single operator library may be, for example, an explicit call to the operator combination library and / or the single operator library to execute a target operator combination and / or a single operator.
[0091] It should also be noted that, without calling an operator combination library and / or a single operator library to execute a target operator combination and / or a single operator, the target operator combination and / or a single operator may be executed by executing a code.
[0092] For example, multiple operator combinations may be maintained through an operator combination library, and the target operator combination is determined by matching from the operator combination library. For example, multiple independent single operators may also be maintained through a single operator library, and the independent single operators of the first operator graph and / or the second operator graph may be determined by matching from the single operator library.
[0093] The following will illustrate specific examples of several operator combinations, which include multiple independent single operators, including sort, sort-merge-join, window, hash-join, hash-aggregation, table-scan, filter, project, and shuffle-exchange, where sort is used for sorting; sort-merge-join is used for sorting, merging, and joining; window is used to calculate all RDDs (RDD is Resilient Distributed Datasets) within a window time; hash-join is used for hash join; hash-aggregation is used to perform aggregation operations on each group after grouping by hash value; table-scan is used for table scanning; filter is used for filtering and screening; project is used to rearrange the parameters and specify different subscripts from the element array of the data stream to return a new data stream; shuffle-exchange is used for shuffling and exchanging.
[0094] 1) An operator combination consisting of two sorts and a sort-merge-join.
[0095] 2) Operator combination consisting of multiple consecutive joins (or hash-join or sort-merge-join).
[0096] 3) Operator combination consisting of sort and window.
[0097] 4) Operator combination consisting of hash-join and hash-aggregation.
[0098] 5) Operator combination consisting of table-scan and filter.
[0099] 6) Operator combination consisting of table-scan and project.
[0100] 7) Operator combination consisting of shuffle-exchange and filter.
[0101] 8) Operator combination consisting of shuffle-exchange and project.
[0102] According to a task execution method of another embodiment of the present disclosure, when the task to be processed is a subtask of the target task, the task execution method further includes: dividing the target task into a plurality of subtasks, and determining the execution result of the target task according to the execution result of each subtask.
[0103] Exemplarily, for example, the execution result of each subtask may be merged correspondingly according to the division logic of the target task to determine the execution result of the target task.
[0104] For example, in a distributed scenario, each subtask can be executed on one or more nodes. Of course, multiple subtasks can also be executed on one node.
[0105] In some cases, for complex target tasks or for reasons such as execution efficiency, the target task can be divided into multiple subtasks. By executing multiple subtasks and merging the processing results of multiple subtasks, the target task can be executed more efficiently and can also adapt to a wider range of application scenarios.
[0106] Exemplarily, the task to be processed or the target task is a query task.
[0107] With the development of computer technology, all aspects of production and life are automated through various applications. The query function is a basic function provided by each application. In addition, under the current big data trend, the amount of data in query tasks is growing exponentially. Through the task execution method of the embodiment of the present disclosure, query tasks can be efficiently executed.
[0108] The following will take a query task in a big data scenario as an example to illustrate a task execution method according to another embodiment of the present disclosure.
[0109] Big data can be understood as a collection of data that is so large that it exceeds the capabilities of traditional database software tools in terms of acquisition, storage, management, and analysis. It has four major characteristics: massive data size, fast data flow, diverse data types, and low value density. Task execution in big data scenarios requires higher execution efficiency.
[0110] Figure 3 A schematic diagram schematically shows a task execution method according to yet another embodiment of the present disclosure.
[0111] exist Figure 3 In the example, take the query task as the target task. The query task is triggered by the query request query. By converting the query task into an operator graph structure, the operator graph of the query task can be obtained (in Figure 3 In the example shown, task execution also includes, for example, execution plan generation, and the generated execution plan is, for example, executed by an executor). The operator graph of the query task enters the execution planner to start generating an execution plan, for example, initializing the current execution plan to be empty, and then dividing the operator graph of the query task into multiple stages (stage, a stage can be understood as a subtask), and then operator combination matching can be performed for each stage, that is, matching the target operator combination from the operator combination library, and for a single operator that cannot be matched with an operator combination in the operator combination library, a single operator matching can be performed based on a single operator library, thereby generating a second operator graph corresponding to each subtask as an execution plan, and the execution plan can enter the executor for execution, thereby outputting the execution result of the query task.
[0112] It should be noted that the above-mentioned execution plan generation and executor can be provided by a task processing framework, which can be a big data processing framework. In the task execution method of the embodiment of the present disclosure, the task processing framework can also provide an operator planning module, which provides the function of operator combination matching. On this basis, the operator planning module can also provide the function of single operator matching.
[0113] Figure 4A A schematic diagram of the first operator graph of stage S is schematically shown. Figure 4B A schematic diagram of the second operator graph of stage S is schematically shown.
[0114] exist Figure 4A In the example, the first operator graph of stage S is schematically shown. There are three input data, namely input data In1, input data In2 and input data In3. Input data In1 passes through operator Op1, operator Op3, operator Op6 and operator Op7 in sequence. Input data In2 passes through operator Op2, operator Op4, operator Op6 and operator Op7 in sequence. Input data In3 passes through operator Op5 and operator Op7 in sequence.
[0115] For example, if there is an operator combination that matches the first operator subgraph of the first operator graph consisting of operators Op3, Op4, and Op6, then the operator combination can be used as the target operator combination OC to replace the corresponding first operator subgraph to obtain the second operator graph. Figure 4BIn the example, a specific example is schematically shown in which the three operators included in the target operator combination OC are operator Op3-1, operator Op4-1, and operator Op6-1.
[0116] In this target operator combination, the data between any two adjacent nodes does not need to be cached. For example, the operator that indicates the data flow direction can be used with an 8-byte index. Figure 4A and Figure 4B In the example, data transmitted between other independent operators needs to be cached. Figure 4A and Figure 4B In the example, a specific example is schematically shown in which operators Op1 and Op2 are Filter&Project, operators Op3 and Op4 are ColumnarSort, operator Op3-1 is SortLeft, operator Op4-1 is SortRight, operator Op5 is Hash-Aggregation, operators Op6 and Op6-1 are Sort-Merge-Join, and operator Op7 is Hash-Join.
[0117] In summary, according to the task execution method of the embodiment of the present disclosure, for example, taking the full-stage code generation as an example, which combines the "chain calls" between operators into one code, since the full-stage code generation uses row-based queries, the computational efficiency of row-based queries is lower than the column-based queries supported by the task execution method of the embodiment of the present disclosure, and there is little room for operator optimization. In addition, for example, taking the operator vectorized execution as an example, which is split into independent single operators as units, the task execution method of operator vectorized execution brings performance losses in data transfer and software engineering encapsulation between operators. The task execution efficiency and performance of the task execution method of the embodiment of the present disclosure are better.
[0118] Figure 5 The block diagram schematically shows a task execution device according to an embodiment of the present disclosure.
[0119] like Figure 5 As shown, the task execution device 500 of the embodiment of the present disclosure includes, for example: a target operator combination determination module 510 , a second operator graph determination module 520 , and an execution module 530 .
[0120] The target operator combination determination module 510 is used to determine, according to the first operator graph corresponding to the task to be processed, a target operator combination that at least partially matches the first operator graph.
[0121] The target operator combination includes multiple operators.
[0122] The second operator graph determination module is used to determine the second operator graph corresponding to the task to be processed according to the target operator combination and the first operator graph.
[0123] The execution module is used to execute the second operator graph based on the received task data to be processed to obtain the execution result of the task to be processed.
[0124] According to a task execution device of yet another embodiment of the present disclosure, the target operator combination determination module includes: a candidate operator combination determination submodule and a target operator combination determination submodule.
[0125] The candidate operator combination determination submodule is used to take each node of the first operator graph as a node to be matched, and determine the operator combination including the node to be matched as a candidate operator combination.
[0126] The target operator combination determination submodule is used to match the candidate operator combination with the first operator subgraph to determine the target operator combination.
[0127] The first operator subgraph is a subgraph of the first operator graph, and the first operator subgraph includes nodes to be matched.
[0128] According to a task execution device of another embodiment of the present disclosure, the target operator combination determination submodule includes: a matching unit, which is used to match the candidate operator combination with the first operator subgraph based on the isomorphism of the graph to determine the target operator combination.
[0129] According to a task execution device of another embodiment of the present disclosure, the execution module includes: a calling submodule, which is used to call the operator combination library to execute the target operator combination based on the received task data to be processed and obtain the execution result of the task to be processed.
[0130] The operator combination library includes target operator combinations.
[0131] According to a task execution device of yet another embodiment of the present disclosure, when the task to be processed is a subtask of the target task, the task execution device further includes: a task division module and a merging module.
[0132] The task division module is used to divide the target task into multiple subtasks.
[0133] The merging module is used to determine the execution result of the target task according to the execution result of each subtask.
[0134] It should be understood that Figure 5 The embodiments of the device part of the present disclosure shown are the same or similar to the embodiments of the method part of the present disclosure, and the technical problems solved and the technical effects achieved are also the same or similar, which will not be repeated in this disclosure.
[0135] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a computer-readable storage medium, and a computer program product.
[0136] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. Electronic devices include various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also include various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0137] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0138] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0139] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the task execution method. For example, in some embodiments, the aforementioned method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the task execution method described above may be executed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the task execution method in any other appropriate manner (eg, by means of firmware).
[0140] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0141] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0142] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, a flash memory, or any suitable combination of the foregoing.
[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0144] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0145] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0146] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0147] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A task execution method, characterized in that: include: Determine, according to a first operator graph corresponding to the task to be processed, a target operator combination that at least partially matches the first operator graph, wherein the target operator combination includes a plurality of operators; Determining a second operator graph corresponding to the task to be processed according to the target operator combination and the first operator graph; Based on the received task data to be processed, the second operator graph is executed to obtain an execution result of the task to be processed.
2. The method according to claim 1, characterized in that in, The determining, according to a first operator graph corresponding to the task to be processed, a target operator combination that at least partially matches the first operator graph comprises: Taking each node of the first operator graph as a node to be matched, determining an operator combination including the node to be matched as a candidate operator combination; The candidate operator combination is matched with a first operator subgraph to determine the target operator combination, wherein the first operator subgraph is a subgraph of the first operator graph, and the first operator subgraph includes the node to be matched.
3. The method according to claim 2, characterized in that in, The matching the candidate operator combination with the first operator subgraph to determine the target operator combination includes: The candidate operator combination is matched with the first operator subgraph based on graph isomorphism to determine the target operator combination.
4. The method according to claim 1, characterized in that: in, The executing the second operator graph based on the received task data to be processed to obtain the execution result of the task to be processed includes: The operator combination library is called to execute the target operator combination based on the received task data to be processed and obtain the execution result of the task to be processed, wherein the operator combination library includes the target operator combination.
5. The method according to claim 1, characterized in that in, When the task to be processed is a subtask of the target task, the method further includes: Dividing the target task into a plurality of subtasks; The execution result of the target task is determined according to the execution result of each subtask.
6. The method according to claim 1 or 5, characterized in that: in, The to-be-processed task or the target task is a query task.
7. A task execution device, characterized in that: include: A target operator combination determination module, configured to determine, according to a first operator graph corresponding to a task to be processed, a target operator combination that at least partially matches the first operator graph, wherein the target operator combination includes a plurality of operators; A second operator graph determination module, configured to determine a second operator graph corresponding to the task to be processed according to the target operator combination and the first operator graph; An execution module is used to execute the second operator graph based on the received task data to be processed to obtain the execution result of the task to be processed.
8. The device according to claim 7, characterized in that in, The target operator combination determination module includes: A candidate operator combination determination submodule, configured to take each node of the first operator graph as a node to be matched, and determine an operator combination including the node to be matched as a candidate operator combination; The target operator combination determination submodule is used to match the candidate operator combination with the first operator subgraph to determine the target operator combination, wherein the first operator subgraph is a subgraph of the first operator graph, and the first operator subgraph includes the node to be matched.
9. The device according to claim 8, characterized in that in, The target operator combination determination submodule includes: A matching unit is used to match the candidate operator combination with the first operator subgraph based on graph isomorphism to determine the target operator combination.
10. The device according to claim 7, characterized in that in, The execution module includes: The calling submodule is used to call the operator combination library to execute the target operator combination based on the received task data to be processed and obtain the execution result of the task to be processed, wherein the operator combination library includes the target operator combination.
11. The device according to claim 7, characterized in that in, When the task to be processed is a subtask of the target task, the device further includes: A task division module, used to divide the target task into multiple subtasks; The merging module is used to determine the execution result of the target task according to the execution result of each subtask.
12. An electronic device, characterized in that: include: At least one processor The at least one processor is used to execute instructions so that the electronic device can execute the method according to any one of claims 1 to 6.
13. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed, the computer is caused to perform the method according to any one of claims 1 to 6.