Data processing method and system, electronic equipment, storage medium and program product

Abstract

The invention discloses a data processing method and system, electronic equipment, a storage medium and a program product. The method comprises the following steps: receiving a data processing task; analyzing the data processing task into a plurality of sub-tasks; based on the constraint of a preset limit rule, allocating an execution engine unit for each subtask in the multiple classes of execution engine units; the multiple types of execution engine units comprise a first type of execution engine unit; the preset limiting rule is used for limiting whether the first-class execution engine unit supports processing of task nodes in all subtasks or not; the task nodes comprise abstract syntax tree nodes or function implementation nodes; and distributing the sub-tasks to the distributed execution engine units, so that the corresponding execution engine units execute the distributed sub-tasks. According to the method and the device, the first type of execution engine units with incomplete task processing capability can be compatible in the data processing system to execute tasks, so that the online time of the first type of execution engine units from development to use is shortened.

Description

Technical Field

[0001] This application relates to the field of large model technology, specifically to a data processing method, system, electronic device, storage medium, and program product. Background Technology

[0002] The execution engine is the core of a distributed Online Analytical Processing (OLAP) system. Currently, distributed OLAP systems use only one type of execution engine. Despite continuous code iterations, the poor performance of the original execution engine units cannot be fundamentally resolved. However, updating the execution engine in data processing systems such as distributed OLAP systems requires testing to ensure the newly developed execution engine fully supports the existing services, then taking the data processing system offline, and replacing all existing execution engine units with the new type. This development-to-deployment cycle is lengthy. Summary of the Invention

[0003] This application addresses the shortcomings of the prior art by proposing a data processing method, system, electronic device, storage medium, and program product. By limiting the task processing capability of a first type of execution engine unit through preset limiting rules, and by constraining the specific task allocation method based on the preset limiting rules when allocating specific data processing tasks, the data processing system can be compatible with the execution of tasks by the first type of execution engine unit with incomplete task processing capability, thereby shortening the time from development to deployment of the first type of execution engine unit.

[0004] A first aspect of this application proposes a data processing method, the method comprising:

[0005] Receive data processing tasks;

[0006] The data processing task is divided into multiple sub-tasks;

[0007] Based on the constraints of preset limiting rules, execution engine units are assigned to each subtask among multiple types of execution engine units; the multiple types of execution engine units include a first type of execution engine unit; the preset limiting rules are used to limit whether the first type of execution engine unit supports processing task nodes in each subtask; the task nodes include abstract syntax tree nodes or function implementation nodes;

[0008] Each subtask is distributed to the assigned execution engine unit, so that the corresponding execution engine unit executes the assigned subtask.

[0009] A second aspect of this application proposes a data processing system, which includes multiple types of execution engine units and a task coordinator;

[0010] The multiple types of execution engine units include the first type of execution engine unit;

[0011] The task coordinator is used to receive data processing tasks; parse the data processing tasks into multiple subtasks; allocate execution engine units to each subtask among the multiple types of execution engine units based on the constraints of preset limiting rules; the preset limiting rules are used to limit whether the first type of execution engine unit supports processing the task nodes in each subtask; the task nodes include abstract syntax tree nodes or function implementation nodes; and distribute each subtask to the allocated execution engine unit so that the corresponding execution engine unit executes the allocated subtask.

[0012] A third aspect of this application proposes a distributed online analytical processing system, including a task coordinator, multiple types of execution engine units, and a database; the multiple types of execution engine units include a first type of execution engine unit, which is a newly added type of execution engine unit in the distributed online analytical processing system;

[0013] The task coordinator is used to receive data processing tasks sent by the client for a data source, and parse the data processing tasks into multiple subtasks; based on the constraints of preset limiting rules, it assigns execution engine units to each subtask among the multiple types of execution engine units, the preset limiting rules being used to limit whether the first type of execution engine unit supports processing the task nodes in each subtask; the task nodes include abstract syntax tree nodes or function implementation nodes; it distributes each subtask to the assigned execution engine unit; the execution engine unit assigned to each subtask in the data processing task is used to receive and execute the subtasks sent by the task coordinator, and collaboratively implement the data processing task, including feeding back the processing results to the client based on the processing of data in the data source.

[0014] A fourth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method described in the first aspect above.

[0015] The fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the first aspect above.

[0016] A sixth aspect of this application provides a computer program product including a computer program that is executed by a processor to implement the method described in the first aspect above.

[0017] Based on the data processing method described in the first aspect above, this application has at least the following beneficial effects or advantages:

[0018] After receiving a data processing task and parsing it into multiple subtasks, based on preset constraints, execution engine units are assigned to each subtask from among multiple types of execution engine units, including the first type of execution engine unit, to complete the data processing task. Preset constraints can limit whether the first type of execution engine unit supports processing the task nodes in each subtask, thereby constraining the specific task allocation method. This allows the first type of execution engine unit with incomplete task processing capabilities to execute tasks within the data processing system. This ensures the stability of online service processing while shortening the time from development to deployment of the first type of execution engine unit, without interrupting online services, thus improving the user experience. New types of execution engine units can be added online, thereby gradually updating the execution engine units.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram illustrating the structure of a data processing system according to some exemplary embodiments;

[0022] Figure 2 This is a schematic diagram illustrating the structure of a data processing system according to some exemplary embodiments;

[0023] Figure 3 This is a schematic diagram illustrating the structure of a data processing system according to some exemplary embodiments;

[0024] Figure 4 This is a flowchart illustrating a data processing method according to some exemplary embodiments;

[0025] Figure 5 This is a schematic diagram illustrating the structure of a data processing apparatus according to an exemplary embodiment;

[0026] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0028] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination," etc.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0031] For some online data processing services, a large number of servers need to be deployed, and customers have high real-time requirements. Distributed analytical databases, with their good system resource scalability, can meet the needs of applications with large data volumes and good real-time performance. For example, Log Service (SLS) is widely used and can support various internal enterprise services and external customer needs. However, current data processing systems typically use only one execution engine. If an execution engine needs to be updated, the data processing system must be taken offline after testing to ensure that the newly developed execution engine fully supports the original service, and all the original execution engine units must be replaced with the new type of execution engine unit. This development and deployment cycle is lengthy.

[0032] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0033] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application, such as... Figure 1 As shown, the data processing system includes multiple execution engine units 101 and a task coordinator 102.

[0034] like Figure 1 As shown, the multi-type execution engine unit 101 includes a first type of execution engine unit 1011. Each type of execution engine unit includes at least one execution engine unit.

[0035] The task coordinator 102 can be used to execute the data processing method provided in the embodiments of this application, specifically including the following steps: receiving a data processing task, parsing the data processing task into multiple subtasks, allocating execution engine units to each subtask among multiple types of execution engine units 101 based on the constraints of preset limiting rules, and distributing each subtask to the allocated execution engine unit so that the corresponding execution engine unit executes the allocated subtask. Each execution engine unit is a unit used to execute a specific subtask. One execution engine unit can be deployed in one or more devices, and multiple execution engine units may also be deployed in one device. The execution engine unit can be an execution engine or a computing engine to realize operations such as querying, calculating, and analyzing data. The data processing task may include multiple operations in serial or parallel order.

[0036] The preset constraint rules are used to limit whether the first type of execution engine unit 1011 supports processing task nodes in each subtask. Any subtask may include at least one type of task node, and each type of task node may include at least one; that is, for a subtask, it may include multiple task nodes of the same type. The type of task node can be an Abstract Syntax Tree (AST) node or a function implementation node.

[0037] In this application embodiment, the data processing system can be a system for processing database data. In some embodiments, the data processing system can be a system capable of processing data based on multiple data sources. Optional examples include systems capable of processing online services, such as distributed OLAP systems and distributed online transaction processing (OLTP) systems. The data processing tasks received by the data processing system can be generated based on data processing requests sent by users of the data processing system. These data processing requests can be generated based on Structured Query Language (SQL).

[0038] In an example application scenario, a user wants the data processing system to combine partial data from database A and partial data from database B to query results that meet the user's set conditions. The user can enter an SQL statement through an input dialog box on the web page provided by the data processing system. The web page generates a data processing request and sends it to the data processing system deployed in the cloud. After receiving the data processing request, the task coordinator 102 of the data processing system generates a data processing task based on the SQL statement, parses the task to obtain each subtask, and allocates each subtask to multiple execution engine units 101. The allocated execution engine units execute each subtask respectively, thereby completing the data processing task and feeding back the results that the user wants to obtain to the web page for display.

[0039] It is understood that in the data processing system provided in this application embodiment, the deployment method of the multiple execution engine units 101 and the task coordinator 102 can be arbitrary. For example, some execution engine units can be deployed in a local online server, and some execution engine units can be deployed in a cloud service; in some embodiments, some execution engine units can be deployed in the task coordinator 102 and / or the data source server; in some embodiments, the task coordinator 102 and each execution engine unit can communicate through corresponding network connection methods, such as a local area network, an internal private network, the Internet, etc.; in some embodiments, different components can be set in the task coordinator 102 to achieve different functions. For example, the task coordinator 102 may include a syntax analysis component and a task scheduler. The syntax analysis component parses the data processing task into multiple subtasks, and the task scheduler sends the information of each subtask to the corresponding assigned execution engine unit. This application embodiment does not limit the specific deployment method of the data processing system; it can be set according to the specific situation. Optionally, the data processing system provided in this application embodiment may further include a data source or a client.

[0040] The data processing system provided in this application embodiment, after receiving a data processing task and parsing it into multiple subtasks through a task coordinator 102, allocates execution engine units to each subtask based on preset constraints and multiple execution engine units, including a first type of execution engine unit. This allows the execution engine units assigned to subtasks within the multiple execution engine units 101 to execute the corresponding subtasks, thereby completing the data processing task. The data processing system provided in this application embodiment can limit whether the first type of execution engine unit supports processing task nodes in each subtask through preset constraints and rules, thus constraining the specific task allocation method. This allows the system to accommodate first type execution engine units with incomplete task processing capabilities, thereby ensuring the stability of online service processing while shortening the update cycle of the data processing system without interrupting online services, improving user experience. New types of execution engine units can be added online, gradually updating the execution engine units.

[0041] In some alternative embodiments, the multi-class execution engine unit 101 includes, for example: Figure 1 The second type of execution engine unit 1012 is shown, while the first type of execution engine unit 1011 is a newly added type of execution engine unit in the data processing system relative to the second type of execution engine unit 1012. That is, the original execution engine units in the data processing system include the second type of execution engine unit 1012, while the new execution engine units include the first type of execution engine unit 1011. By limiting the processing capabilities of the task nodes of the newly added execution engine units, the ability of the new execution engine unit to handle subtasks can be considered during allocation, thereby avoiding the allocation of subtasks that the new execution engine unit cannot handle to the new execution engine unit, and improving the accuracy and stability of data processing task completion. Optionally, for the same type of task nodes, the execution efficiency / speed of the new execution engine unit is higher than that of the original execution engine unit. Thus, the processing efficiency of the data processing system can be improved after adding the new execution engine unit.

[0042] If the first type of execution engine unit 1011 includes a first execution engine unit, and the multiple subtasks include a first subtask to be sent to the first execution engine unit, in some embodiments, the task coordinator 102 may be configured with a first protocol. The task coordinator 102 can convert the first subtask into first subtask data that the first execution engine unit can correctly parse based on the first protocol. The first execution engine unit can parse the first subtask after receiving the first subtask data. Optionally, the first protocol may include execution protocols for various types of task nodes. Since the task coordinator 102 needs to be compatible with multiple types of execution engine units, and it is the task coordinator 102 that sends data to each execution engine unit, configuring the first protocol on the task coordinator 102 side can facilitate the task coordinator 102 to convert each subtask into subtask data that can be correctly parsed by the corresponding assigned execution engine unit when sending data. At the same time, since the first protocol needs to be compatible with many types of task nodes and the protocol is relatively complex, configuring the first protocol on the task coordinator 102 side also facilitates the updating and iteration of the first type of execution engine unit 1011 itself.

[0043] In some embodiments, the task coordinator 102 may be configured with a second protocol, and the task coordinator 102 may parse the running status statistics sent by the first type of execution engine unit 1011 based on the second protocol.

[0044] Optionally, the running status statistics sent by the first type of execution engine unit 1011 may include periodically reported or reported statistical information such as task execution status, input / output data, execution time, and memory usage based on the instructions of the task coordinator 102. The task coordinator 102 can determine the running status of the execution engine unit based on the running status statistics, and then decide whether to delete the corresponding subtask. For example, if the running status statistics indicate that the corresponding execution engine unit has completed the current subtask, the task coordinator 102 can delete the subtask and set the corresponding execution engine unit as available, placing it in the allocable execution resources of another data processing task to be executed; or, if the running status statistics indicate that the corresponding execution engine unit has not responded for a preset time, the subtask can be deleted, handed over to another execution engine unit for execution, and the allocation of subtasks to the execution engine unit can be suspended until the execution engine unit sends heartbeat information or correct running status information; it is understood that the above are only illustrative examples and are not intended to limit the embodiments of this application.

[0045] In some other embodiments, the first type of execution engine unit 1011 may be configured with a third protocol, and the first type of execution engine unit 1011 may send heartbeat information to the task coordinator 101 based on the third protocol. The heartbeat information sent by the first type of execution engine unit 1011 may include heartbeat information reported periodically or reported based on the request of the task coordinator 102.

[0046] Typically, the protocols of the new and old execution engine units are well compatible for these operational status statistics and heartbeat information. Therefore, the aforementioned related protocols can be configured on the task coordinator 102 side.

[0047] The first type of execution engine unit 1011 may be configured with a fourth protocol. The first type of execution engine unit 1011 can parse the first data sent by the third type of execution engine unit based on the fourth protocol, or generate second data to be sent to the third type of execution engine unit based on the fourth protocol. The first data and the second data are respectively the data that the first type of execution engine unit needs to receive or send to execute the assigned subtask. The third type of execution engine unit is any type of execution engine unit other than the first type of execution engine unit among multiple types of execution engine units. Optionally, the third type of execution engine unit may include the second type of execution engine unit described above.

[0048] That is, the first type of execution engine unit 1011 can communicate with other first type of execution engine units based on the fourth protocol. When different subtasks need to interact with corresponding execution engine units, if the execution engine units are of different types, the fourth protocol can be used to transmit compatible data, thereby improving the accuracy, reliability and stability of task execution.

[0049] Based on the data processing system of the above embodiments, in one application scenario, this application embodiment also provides a distributed online analytical processing (OLAP) system, such as... Figure 2 As shown, the distributed online analytical processing system 20 includes a task coordinator 202, a multi-type execution engine unit 201, and a data source 203.

[0050] The multi-type execution engine unit 201 includes a first type of execution engine unit 2011, which is a newly added type of execution engine unit in the distributed online analytical processing system.

[0051] The task coordinator 202 receives data processing tasks for data source 203 sent by client 204, parses the data processing tasks into multiple subtasks, and then assigns execution engine units to each subtask among multiple types of execution engine units 201 based on preset constraints. Here, the preset constraints are used to limit whether the first type of execution engine unit 2011 supports processing the task nodes in each subtask. Task nodes may include abstract syntax tree nodes or function implementation nodes. After assigning execution engine units to each subtask, the task coordinator 202 distributes each subtask to the assigned execution engine unit, so that each execution engine unit, after receiving the subtask sent by the task coordinator 202, performs operations on data source 203 according to the subtask, such as query operations, update operations, etc. Data source 203 may include multiple data sources, which may include databases, such as relational databases, non-relational databases, graph databases, etc., and may also include other data sources, such as data platforms, other services, etc.

[0052] The execution engine units assigned to each subtask in the data processing task can work together to complete the data processing task, including feeding back the processing results to the client based on the processing of data in data source 203 (such as merging, filtering, joining, etc.).

[0053] A specific example provided is a distributed online analytical processing system such as Figure 3 As shown, the architecture of this distributed online analytical processing (OLAP) system example is first introduced: it includes a task coordinator 32 and multiple execution engine units 33. The task coordinator 32 includes a syntax analysis component 321, a JSON protocol conversion component 322, a task scheduler 323, a data source information management component 324, and a query optimizer 325. The multiple execution engine units 33 specifically include first-type execution engine units 331-335 and second-type execution engine units 336-337.

[0054] This distributed online analytical processing system communicates with client 31 and data sources. The data sources include the log service SLS341 and a Structured Query Language (SQL) database 342.

[0055] The data processing task sent by client 31 is parsed by syntax analysis component 321, resulting in multiple subtasks. JSON protocol conversion component 322 has pre-configured default rules for different abstract syntax tree nodes or function implementation nodes, used to limit whether the first type of execution engine unit supports the corresponding abstract syntax tree node or function implementation node. Each subtask, after conversion by JSON protocol conversion component 322, is sent to task scheduler 323. Thus, task scheduler 323, based on the configuration of JSON protocol conversion component 322, determines whether the subtask nodes contained in each subtask support processing by the first type of execution engine unit. If all are supported, the subtask is allowed to be assigned to the first type of execution engine unit; otherwise, it is not allowed to be assigned.

[0056] Based on the same or similar inventive concepts, embodiments of this application also provide a data processing method, which can be applied to the data processing system provided in embodiments of this application. Specifically, it can be applied to the task coordinator in the data processing system provided in embodiments of this application. Descriptions of terminology, action steps, technical effects, etc., in the data processing method and data processing system provided in embodiments of this application can be used as references to each other, and identical parts will not be repeated.

[0057] Figure 4 A flowchart of the data processing method provided in the embodiments of this application is shown below. Figure 4 As shown, the method specifically includes the following steps 401-404.

[0058] Step 401: Receive data processing task.

[0059] Data processing tasks can be initiated by the client or by other services or callers. The data processing method provided in this application can be applied to a data processing system. For a description of the specific deployment method of the data processing system, please refer to the data processing system provided in this application. It is understood that this application does not limit the specific deployment method of the data processing system.

[0060] A data processing task is a task that processes data from a data source, which can include multiple different data sources. By processing the data in the data sources—for example, querying, deleting, and updating—operations such as merging, joining, and filtering can be performed on the queried data. In one example, a data processing task might include SQL statements based on the SQL language input by the user.

[0061] Step 402: Divide the data processing task into multiple subtasks.

[0062] Upon receiving a data processing task, it is parsed into multiple subtasks. A data processing task can be viewed as a final task, which can be broken down into specific units that can be executed by the execution engine unit, i.e., subtasks. These subtasks can include parallel tasks or sequential tasks. For example, if the first subtask needs to be executed after obtaining data from parallel second and third subtasks, then the first and second subtasks are sequential, and the first and third subtasks are also sequential. In some embodiments, the data processing task is an SQL statement, and each subtask is a plan fragment (PlanFragment) obtained by dividing the SQL statement. The smallest unit for the execution engine unit to execute a task is a plan fragment.

[0063] Step 403: Based on the constraints of preset limiting rules, assign execution engine units to each subtask among multiple types of execution engine units.

[0064] The multiple types of execution engine units include a first type of execution engine unit. Preset constraint rules are used to limit whether the first type of execution engine unit supports processing task nodes in each subtask. In this embodiment, the first type of execution engine unit may not fully support all types of task nodes present in the data processing system. When allocating subtasks, subtasks are allocated based on the constraints of the preset constraint rules. If the first type of execution engine unit does not support processing any task node in a subtask, then the first type of execution engine unit does not support processing that subtask, and that subtask is not allowed to be allocated to the first type of execution engine unit. If the first type of execution engine unit supports processing all task nodes in a subtask, then the first type of execution engine unit supports processing that subtask, and that subtask is allowed to be allocated to the first type of execution engine unit.

[0065] It is understood that for a subtask, even if it is allowed to be assigned to a first type of execution engine unit, it may not be assigned to a first type of execution engine unit due to other allocation principles or strategies. For example, all first type of execution engine units may be occupied to process other subtasks, or the idle rate of other types of execution engine units may be high. This application embodiment does not make specific limitations in this regard. The preset limitation rules are only used to limit whether the first type of execution engine unit supports processing different types of task nodes, and thus limit whether it is allowed to be assigned to the first type of execution engine unit. They are not used to force the allocation to the first type of execution engine unit when the first type of execution engine unit supports processing.

[0066] It should be noted that the processing support described in this embodiment is not determined based on real-time conditions such as the current running status of the first type of execution engine unit, but rather on a global perspective, whether the first type of execution engine unit supports processing task nodes. Specifically, the first type of execution engine unit may not have the ability to process at least one type of task node due to development reasons. Therefore, the processing capability of the first type of execution engine unit can be limited by preset limiting rules.

[0067] Each subtask includes at least one task node. Task nodes have different types; if a subtask includes multiple task nodes, these nodes can be of the same or different types. The type of a task node can be an abstract syntax tree node or a function implementation node. An abstract syntax tree node includes expressions, and expressions include functions. However, functions can be categorized into different function implementation nodes based on differences in function name, number of parameters, parameter types, and parameter content. In some embodiments, the types of function implementation nodes may include more than 2000. Therefore, the preset limiting rules can be used to classify and discuss different abstract syntax tree nodes, or more subdivided to classify and discuss different function implementation nodes. It is understood that, in addition to the aforementioned abstract syntax tree nodes or function implementation nodes, the types of task nodes can be further subdivided into other nodes within the abstract syntax tree, thereby allowing for finer-grained limitation of the processing capabilities of the first type of execution engine unit.

[0068] Step 404: Distribute each subtask to the assigned execution engine unit so that the corresponding execution engine unit executes the assigned subtask.

[0069] After allocating subtasks based on preset constraints, each subtask is distributed to its corresponding execution engine unit. Upon receiving a subtask, each execution engine unit executes it according to its assigned content and execution order. For example, if an execution engine unit receives a subtask that requires data from other execution engine units, it waits to receive the data, processes it, and then sends the processing result to the execution engine unit corresponding to the next stage subtask. Here, the order in which execution engine units the data is received from and sent to can be pre-defined in the distributed subtasks. This application does not limit the specific implementation method; for example, execution engine units can be distinguished and identified using their addresses, numbers, or other information.

[0070] Currently, data processing systems typically only have one type of execution engine unit. This application supports the mixed use of multiple types of execution engine units, increasing the variety of execution engine unit types in the data processing system and facilitating the execution of different subtasks using the characteristics of different execution engine units. This application can be applied to updating execution engine units. In some embodiments, the first type of execution engine unit can be the newest type in the data processing system, or it can be a newly added type of execution engine unit relative to at least one other type. This allows for the addition of execution engine units with incomplete functionality to the data processing system, enabling the rapid deployment of newly developed execution engine units and facilitating faster problem identification and adjustments during actual use, shortening the update cycle from development to deployment. Furthermore, it allows for the mixed use of existing execution engine units with more comprehensive functionality, ensuring the stability of online services. Mixing newly added and existing types of execution engine units does not require interrupting online services for updates, remaining seamless for users and improving user experience. Existing execution engine units in the data processing system can be gradually reduced and eventually decommissioned, enabling gradual updates of execution engine units.

[0071] In some embodiments, for the same type of task nodes, the newly added first type of execution engine unit has higher processing efficiency than the existing other types of execution engine units, resulting in higher processing efficiency of the data processing system. In one embodiment, the first type of execution engine unit is based on a first compiled language, while other types of execution engine units are based on different compiled languages. The first compiled language used by the first type of execution engine unit makes its processing efficiency higher. For example, the first type of execution engine unit is written in C++, and the second type of execution engine unit is written in Java.

[0072] In some embodiments, step 403, based on the constraints of preset limiting rules, assigns execution engine units to each subtask among multiple types of execution engine units, and may include the following steps A1 to A3:

[0073] Step A1: Based on the constraints of preset limiting rules, determine whether the first type of execution engine unit supports processing each task node in the first subtask. The first subtask is any one of multiple subtasks.

[0074] Step A2: If the first type of execution engine unit does not support processing any task node in the first subtask, the first subtask is assigned to an execution engine unit other than the first type of execution engine unit among the multiple types of execution engine units.

[0075] Step A3: If the first type of execution engine unit supports processing each task node in the first subtask, the first subtask is assigned to the first type of execution engine unit.

[0076] Based on steps A1 to A3, when allocating subtasks to multiple types of execution engine units, the following allocation strategy is provided: For any subtask, first consider whether it can be allocated to the first type of execution engine unit. Only if the first type of execution engine unit does not support processing is it allocated to other types of execution engine units. Of course, in some embodiments, the idle status of the first type of execution engine unit can also be considered when allocating. For example, if there is no idle first type of execution engine unit, but there are idle execution engine units of other types, then it can also be allocated to other types of execution engine units. Through this allocation strategy, the data processing method provided in this application embodiment can allocate as many subtasks as possible to the first type of execution engine unit, based on the fact that the first type of execution engine unit supports processing. This allows for greater use of the first type of execution engine unit to process tasks, which is beneficial for testing the functionality of the first type of execution engine unit. Furthermore, if the processing efficiency of the first type of execution engine unit is higher, the processing efficiency of the data processing task can also be improved.

[0077] In some embodiments, step 403 determines whether the first type of execution engine unit supports processing each task node in the first subtask based on the constraints of preset limiting rules. Specifically, it can determine whether the first type of execution engine unit supports processing each task node in the first subtask based on the target attributes of each task node. The target attributes of each task node are pre-configured for the task node type to which each task node belongs. The target attributes are used to indicate the preset limiting rules. That is, a specific implementation of the preset limiting rules can be to add attributes to various types of task nodes to describe whether the first type of execution engine unit supports that type of task node. Specifically, various types of task nodes can include at least one attribute. A target attribute is added to various types of task nodes, and the value of the target attribute is used to indicate whether the first type of execution engine unit supports processing that type of task node. For example, if the value of the target attribute is 1, it indicates support; if it is 0, it indicates no support. It is understood that in some specific embodiments, target attributes can be added for some types of task nodes, and task nodes that do not contain target attributes can be defaulted to no support.

[0078] Each task node can be either an abstract syntax tree (AST) node or a function implementation node. An AST node is a node in the abstract syntax tree obtained after the SQL statement has been transformed. Each subtask can include at least one AST node.

[0079] In some embodiments, an abstract syntax tree node may include at least one of the following types: a logical plan node (PlanNode), an expression (Expression), and a data source (PageSource).

[0080] The PlanNode for a newly added target attribute can include at least one of the following types: TableScanNode, ProjectNode, FilterNode, AggregationNode, GroupIdNode, LimitNode, DistinctLimitNode, MarkDistinctNode, SortNode, TopNNode, JoinNode, SemiJoinNode, WindowNode, TopNRowNumberNode, RowNumberNode, OffsetNode, ValuesNode, UnionNode, IntersectNode, ExceptNode, UnnestNode, AssignUniqueIdNode, EnforceSingleNode, OutputNode, ExchangeNode, and RemoteSourceNode.

[0081] The Expression for a newly added target property can include at least one of the following types: VariableReferenceExpression, InputReferenceExpression, ConstantExpression, CallExpression, and SpecialFormExpression.

[0082] The newly added target attribute's data source PageSource can include scheduling units corresponding to various different data source types. For example, the data source type can include at least one of the following types: mysql (a relational database management system), Redis (Remote Dictionary Server, an open-source in-memory data structure storage system), Kafka (an open-source distributed streaming media platform used to build real-time data pipelines and streaming applications), etc.

[0083] The aforementioned CallExpression is also known as a function call or function. In some embodiments, functions can be divided into three categories: scalar functions, aggregate functions, and window functions. The same function name may correspond to multiple different function implementation nodes depending on the number, type, and content of the parameters. Furthermore, the first type of execution engine unit may only support a subset of function implementation nodes for a given function name, depending on the number, type, and content of the parameters. For example, the newly added first type of execution engine unit only supports a few common regular expression matching syntaxes for regular expression functions.

[0084] Since the number of function implementation nodes is quite large, reaching around 2000, preset constraint rules can be set for various function implementation nodes in the function call based on differences in function name, parameter type, and parameter content. That is, target attributes can be added to each function implementation node to achieve finer-grained constraint. Specifically, parameter type is mainly categorized based on the number and type of function parameters, while parameter content is mainly categorized based on the constant values ​​of a specific function parameter.

[0085] By adding target attributes, various task nodes come with their own attributes to represent preset constraint rules. This makes it easier for the executor of the data processing method provided in this application to determine the target attributes of each task node in each subtask after generating subtasks, eliminating the need to call other rule information and improving allocation efficiency.

[0086] In some embodiments, step 404 distributes each subtask to an assigned execution engine unit, which may specifically include steps B1 to B2:

[0087] Step B1: Based on the pre-configured target protocol, convert the second subtask into second subtask data.

[0088] The second subtask is the subtask among multiple subtasks to be distributed to the first execution engine unit. The target protocol is used to enable the first execution engine unit to parse the second subtask data and obtain the second subtask. The first execution engine unit is a first type of execution engine unit allocated to the second subtask. In some embodiments, the target protocol may be the first protocol as described in the data processing system provided in the embodiments of this application.

[0089] Step B2: Distribute the data of the second subtask to the first execution engine unit.

[0090] Because it is compatible with multiple types of execution engine units, the support of a target protocol is required so that the first type of execution engine unit can correctly parse the received subtask data to obtain the subtask. In some embodiments, the target protocol or the first protocol is the execution protocol, meaning that the first type of execution engine unit can correctly parse the received subtask data.

[0091] Furthermore, execution protocols can be configured separately for each type of task node supported by the first type of execution engine unit. The task node type can be an abstract syntax tree node or a function implementation node; specific details can be found in the node types executed in the specific embodiments for configuring target attributes, and will not be repeated here. Because compatibility with each type of task node is required, the target protocol configuration is relatively complex. Configuring it on the executor of the data processing method provided in this application embodiment, rather than on the side of the first type of execution engine unit, facilitates the updating and iteration of the first type of execution engine unit.

[0092] In some embodiments, step 403, based on the constraints of preset limiting rules, allocates execution engine units to each subtask among multiple types of execution engine units, and may include steps C1 to C2:

[0093] Step C1: Based on the running status information of the first type of execution engine unit, determine whether to suspend the use of the first type of execution engine unit;

[0094] Step C2: Based on the constraints of the preset limiting rules and the judgment result of whether to suspend the use of the first type of execution engine unit, an execution engine unit is assigned to each subtask among the multiple types of execution engine units.

[0095] The running status information is used to indicate the running status of the first type of execution engine unit. Optionally, it may include information on the current running status, historical running status, the running status of each execution engine unit, or the overall running status of the first type of execution engine unit. For example, the running status information may include whether each first type of execution engine unit is currently executing a subtask and its execution duration, processing progress, input data, output data, etc. It may also include whether each first type of execution engine unit has subtasks waiting to be executed and the number of subtasks, or it may include the execution duration and whether errors occurred for each type of subtask. Based on the running status information, it can be determined whether the use of the first type of execution engine unit needs to be suspended. For example, if the error rate of the first type of execution engine unit when executing a certain type of task node is higher than a preset threshold, the use of the first type of execution engine unit to process that type of task node will be suspended. Similarly, if all first type of execution engine units are executing subtasks, or if each first type of execution engine unit has a large number of subtasks waiting to be processed, the allocation of subtasks to the first type of execution engine unit will be suspended. The above are only examples. By monitoring the operational status of the first type of execution engine unit through operational status information, we can respond promptly to problems that occur during the actual operation of the first type of execution engine unit, suspend its use, improve the online stability of the data processing system, and more smoothly cope with online emergencies.

[0096] For step C1: Based on the running status information of the first type of execution engine unit, determine whether to suspend the use of the first type of execution engine unit. In some specific embodiments, based on the running status information of the first type of execution engine unit, determine whether to suspend the use of the first type of execution engine unit for a target object. The target object includes at least one of the following: target user, target data processing task, and target task node. In this way, the first type of execution engine unit can be enabled or disabled for different target objects.

[0097] In some embodiments, step C1 above can set a system-level switch for the entire data processing system to temporarily enable or disable the first type of execution engine unit. Furthermore, switches can be set separately for different users, different data processing tasks, and different types of task nodes to enable or disable the first type of execution engine unit.

[0098] In other embodiments, a blacklist can be set for the first type of execution engine unit to place certain users, certain types of task nodes, or certain data processing tasks (e.g., data processing tasks currently received or received within a certain period of time) into the blacklist, prohibiting the allocation of subtasks involving or including the corresponding target object to the first type of execution engine unit.

[0099] Based on different target object classifications, it is possible to determine whether to suspend the use of the first type of execution engine unit. This allows for hierarchical control over the objects that the first type of execution engine unit is enabled for, thus improving the flexibility of dividing the scope of suspension.

[0100] The following is combined Figure 3 This paper describes the implementation process of a data processing method provided in this application embodiment within a distributed online analytical processing (OLAP) system. For Figure 3 The specific structure of the distributed OLAP system shown can be found in the documentation on distributed OLAP systems. Figure 3 The explanation will not be repeated here.

[0101] In this embodiment, the first type of execution engine unit is a newly added type of execution engine unit relative to the second type of execution engine unit, while the second type of execution engine unit is an existing type of execution engine unit. Therefore, compared to the first type of execution engine unit, the second type of execution engine unit has more comprehensive support functions for each task node, and the communication protocol between the task coordinator 32 and the second type of execution engine unit is compatible.

[0102] In this embodiment, multiple protocols can be pre-configured in the task coordinator 32 to ensure that the newly added first type of execution engine unit is compatible with communication between the second type of execution engine unit and the task coordinator 32. Specifically, the protocols may include control flow compatibility and data flow compatibility.

[0103] Control flow compatibility is used to ensure compatibility of control flow data between the task coordinator 32 and the first type of execution engine unit. Specifically, the protocol may include: an execution protocol, a heartbeat protocol, and a runtime statistics protocol.

[0104] The execution protocol enables the first type of execution engine unit to correctly parse the subtask data sent by the task coordinator 32 to obtain the correct subtask. Execution protocol compatibility can include the configuration first protocol described in the distributed OLAP system provided in this application embodiment, or the configuration target protocol described in the data processing method provided in this application embodiment. If the task coordinator 32 schedules a subtask to a first type of execution engine unit, it can generate corresponding subtask data based on the execution protocol and distribute the subtask data to the corresponding first type of execution engine unit. Execution protocol compatibility is relatively complex, requiring compatibility with various task nodes, including logical plan nodes (PlanNode), expressions (Expression), and data sources (PageSource) in the abstract syntax tree. Since the first type of execution engine unit is newly added and iterates quickly, execution protocol compatibility can be implemented on the task coordinator 32 side. Examples of specific compatible task nodes can be found in the above description of the specific implementation of the target attribute, and will not be repeated here.

[0105] A heartbeat protocol is used to enable the task coordinator 32 to correctly parse heartbeat information sent by the execution engine units. The heartbeat information informs the execution engine units of their operational health. Heartbeat protocol compatibility may include the configuration third protocol described in the distributed OLAP system provided in this application embodiment. Each execution engine unit may periodically submit heartbeat information to the task coordinator 32 or based on requests from the task coordinator 32. The task coordinator 32 can determine whether the execution engine unit is operating normally based on the heartbeat information; if no heartbeat information is received for a preset time, the allocation of subtasks to the corresponding execution engine unit is suspended. The heartbeat protocol can be configured on the first type of execution engine side to achieve compatibility between the heartbeat protocol and the task coordinator 32 and the second type of execution engine units. The task coordinator 32 can parse the heartbeat information sent by the first type of execution engine unit according to the previous heartbeat protocol.

[0106] The runtime status statistics protocol enables the task coordinator 32 to correctly parse the runtime status statistics information sent by the execution engine units. Compatibility of the runtime status statistics protocol can include the configuration second protocol described in the distributed OLAP system provided in this application embodiment. The runtime status statistics information can represent the execution status, input / output, execution time, memory usage, and other statistical information of subtasks. Each execution engine unit can periodically submit runtime status statistics information to the task coordinator 32, allowing the task coordinator 32 to determine whether to allocate subtasks based on the runtime status of each execution engine unit. In some embodiments, the task coordinator 32 determines whether to delete the corresponding task based on the runtime status of the execution engine unit. For example, if a data processing task is completed, the data processing task can be deleted, and the corresponding execution engine unit can be marked as idle; if any subtask fails, the corresponding subtask can be deleted and restarted through other execution engine units, and the erroneous execution engine unit can be marked as temporarily unusable. The runtime status statistics protocol is relatively easy to be compatible with. The runtime status statistics protocol can be configured on the task coordinator side to achieve compatibility between new and old execution engine units.

[0107] Data flow compatibility is used to ensure compatibility between the task coordinator 32 and the first type of execution engine unit and / or the first type of execution engine unit and the second type of execution engine unit. Data flow compatibility may include the configuration of the fourth protocol described in the distributed OLAP system provided in this application embodiment, with the aim of ensuring compatibility with serialization and deserialization of various data types. The codec type required for the configuration protocol in data flow compatibility may include at least one of the following types: ArrayBlockEncoding, ByteArrayBlockEncoding, DictionaryBlockEncoding, FixedWidthBlockEncoding, IntArrayBlockEncoding, LazyBlockEncoding, LongArrayBlockEncoding, RunLengthBlockEncoding, ShortArrayBlockEncoding, and VariableWidthBlockEncoding.

[0108] In addition, the JSON protocol conversion component 322 needs to be configured with rules limiting the first type of execution engine unit. Specifically, this means limiting whether the first type of execution engine unit is supported for different types of task nodes. Different task nodes can be different abstract syntax tree nodes or different function implementation nodes.

[0109] After configuring the above protocols, the newly added first type of execution engine unit can be compatible with the original second type of execution engine unit and task coordinator 32.

[0110] refer to Figure 3 If client 31 receives a user's operation, for example, the client can be in the form of a webpage, and the user enters an SQL statement in the input box of the webpage, then client 31 sends a data processing task containing the SQL statement to the syntax analysis component 321 based on the user's operation.

[0111] The syntax analysis component 321 is used to parse SQL statements and generate multiple subtasks. Furthermore, the syntax analysis component 321 can also perform semantic verification on the SQL statements to determine whether the SQL statements are logically meaningful and conform to the semantic rules of the database. In this embodiment, the syntax analysis component 321 parses the SQL statement input by the user to generate an abstract syntax tree including multiple abstract syntax tree nodes.

[0112] The query optimizer 325 can find the execution plan with the fastest execution speed and the least execution resource consumption based on the abstract syntax tree. The generated execution plan includes multiple subtasks and execution engine units allocated to each subtask. Each subtask includes at least one task node. The JSON protocol conversion component 322 can serialize each subtask into subtask data to be sent. The task scheduler 233 executes the specific subtask data distribution process. The task scheduler 323 can send the subtask data to each execution engine unit based on the control flow shown by the dashed lines.

[0113] In one example, the first type of execution engine unit does not support processing data source 342. Therefore, a subtask of retrieving data from data source 342 is assigned to the second type of execution engine unit 337. After performing a query operation, the second type of execution engine unit 337 sends the query results to the first type of execution engine units 331-332 respectively. The first type of execution engine units 333-335 query data from data source 341 and send the query results to the first type of execution engine units 331-332 respectively. After performing operations on the received data, the first type of execution engine units 331-332 send the processing results to the second type of execution engine unit 336, which then performs the operation and sends the processing results to the client 31. Figure 3 The solid lines within the multiple execution engine units 33 are used to represent data flow and data transmission direction.

[0114] The task scheduler 323 can obtain heartbeat information and running status statistics based on the control flow shown by the dashed line, and delete the data processing task after determining that the data processing task has been completed.

[0115] The execution engine unit is a core component of a distributed OLAP system. Currently, some general distributed OLAP systems only have one type of execution engine unit. Although the code has been continuously iterated, the problem of poor performance of the original execution engine unit cannot be fundamentally solved.

[0116] This application provides a distributed OLAP system with multiple execution engine units running concurrently. A user-sent SQL statement can be executed jointly by both new and old execution engine units. In some embodiments, the new execution engine unit is written in C++, while the old execution engine unit is written in Java. Since the newly added first type of execution engine unit uses C++, its performance is significantly improved. While ensuring functional consistency and stable operation, using the higher-performance new execution engine unit results in faster execution speed, shortening processing time and improving the overall execution efficiency of the distributed OLAP system. For embodiments where the data source includes log services and log analysis services provided by the data processing system, this can improve the real-time performance of the log analysis service, better meeting the real-time needs of customers.

[0117] The new execution engine unit may lack comprehensive functionality, while the old execution engine unit offers more complete features. After SQL partitioning, the PlanFragment remains unchanged. Each independent PlanFragment should ideally be executed using the new execution engine unit. If the new execution engine unit is limited and cannot support the processing, then the PlanFragment should still be executed using the old execution engine unit. This allows for seamless integration between the new and existing execution engine units without affecting the normal operation of the online distributed OLAP system and ensuring complete functional consistency. Most operators and functions are smoothly transitioned from the old to the new computing engine, gradually improving the execution performance of the online system. If the new execution engine unit can support most online data processing task requests, the processes of the old execution engine unit can be gradually decommissioned, thereby improving SQL execution performance while reducing hardware costs.

[0118] Because the new execution engine unit was developed in a short time and has not been running on the online system for an extended period, its stability still needs continuous improvement. A multi-level system switch can be designed, comprising four levels: system-level, session-level, user-level, and function-level. The system-level switch controls whether the new execution engine unit is enabled across the entire execution engine unit cluster. The session-level switch controls whether the new execution engine unit is enabled for the current data processing task. The user-level switch controls whether the new execution engine unit is enabled for a specific user (if the system is shared by multiple users). The function-level switch controls the blacklist of functions for the new execution engine unit. By setting different levels of switches, the new execution engine unit can be configured to temporarily disable its use for certain objects, allowing for timely responses to unexpected situations online. This rapid and targeted shutdown of the new execution engine unit minimizes the impact on the online system, achieving a gradual, gradual rollout without affecting the overall system, resulting in good stability.

[0119] Corresponding to the embodiments of the aforementioned data processing methods, this application also provides embodiments of a data processing apparatus. Since the inventive concepts are the same or similar, any parts of the data processing apparatus provided in the embodiments of this application that are not described in detail can be referred to the relevant descriptions of the data processing methods or data processing systems provided in the embodiments of this application. In one example, the data processing apparatus provided in the embodiments of this application may be implemented through a combination of hardware modules and / or software units in the task coordinator of the data processing system provided in the embodiments of this application.

[0120] Figure 5 This is a schematic diagram illustrating the structure of a data processing apparatus according to an exemplary embodiment. The apparatus is used to execute the data processing method provided in any of the above embodiments, such as... Figure 5 As shown, the data processing device includes: a receiving module 501, a parsing module 502, an allocation module 503, and a distribution module 504.

[0121] The receiving module 501 is used to receive data processing tasks.

[0122] The parsing module 502 is used to parse the data processing task into multiple sub-tasks.

[0123] The allocation module 503 is used to allocate execution engine units to each subtask among multiple types of execution engine units based on the constraints of preset limiting rules. The multiple types of execution engine units include a first type of execution engine unit; the preset limiting rules are used to limit whether the first type of execution engine unit supports processing task nodes in each subtask; the task node includes an abstract syntax tree node or a function implementation node.

[0124] The distribution module 504 is used to distribute each subtask to the assigned execution engine unit so that the corresponding execution engine unit can execute the assigned subtask.

[0125] In some embodiments, the allocation module 503 may specifically be used to perform:

[0126] Based on the constraints of preset limiting rules, it is determined whether the first type of execution engine unit supports processing each task node in the first subtask; the first subtask is any one of the multiple subtasks.

[0127] If the first type of execution engine unit does not support processing any task node in the first subtask, the first subtask is assigned to an execution engine unit other than the first type of execution engine unit among the multiple types of execution engine units;

[0128] If the first type of execution engine unit supports processing each task node in the first subtask, the first subtask is assigned to the first type of execution engine unit.

[0129] Furthermore, in some embodiments, the allocation module 503 may specifically be used to perform:

[0130] Based on the target attributes of each task node in the first subtask, determine whether the first type of execution engine unit supports processing each task node in the first subtask; wherein, the task node type corresponding to each task node is pre-configured with target attributes for indicating the preset limiting rules.

[0131] In some embodiments, for the same type of task node, the first type of execution engine unit performs processing more efficiently than the other execution engine units among the multiple types of execution engine units.

[0132] In some embodiments, the distribution module 504 is specifically used for:

[0133] Based on a pre-configured target protocol, the second subtask is converted into second subtask data; the second subtask is any one of the plurality of subtasks to be distributed to the first type of execution engine unit; the target protocol is used to enable the first execution engine unit to parse the second subtask data to obtain the second subtask; the first execution engine unit is the first type of execution engine unit allocated for the second subtask.

[0134] The second subtask data is distributed to the first execution engine unit.

[0135] In some embodiments, the allocation module 503 may include a judgment submodule and an allocation submodule.

[0136] The determination submodule is used to determine whether to suspend the use of the first type of execution engine unit based on the running status information of the first type of execution engine unit;

[0137] The allocation submodule is used to allocate execution engine units to each subtask among the multiple types of execution engine units based on the constraints of the preset limiting rules and the judgment result of whether to suspend the use of the first type of execution engine unit.

[0138] In some embodiments, the determination submodule may be specifically used to perform:

[0139] Based on the running status information of the first type of execution engine unit, determine whether to suspend the use of the first type of execution engine unit for the target object; the target object includes at least one of the following: target user, target data processing task, target task node.

[0140] The data processing apparatus and the data processing method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0141] Specifically, currently, only one type of execution engine unit is typically set up in the data processing system. This application embodiment supports the mixed use of multiple types of execution engine units, which increases the types of execution engine units in the data processing system and facilitates the use of the characteristics of different execution engine units to execute different subtasks. This application embodiment can be applied to updating execution engine units. In some embodiments, the first type of execution engine unit can be the newest type of execution engine unit in the data processing system, or it can be a newly added type of execution engine unit relative to at least one other type. This allows for the addition of execution engine units with incomplete functionality to the data processing system, facilitating the rapid deployment of newly developed execution engine units and enabling faster problem identification and adjustments during actual use, shortening the update cycle from development to deployment. Furthermore, it allows for the mixed use of existing execution engine units with more comprehensive functionality, ensuring the stability of online services. Mixing newly added and existing types of execution engine units does not require interrupting online services for updates, resulting in a seamless user experience. Existing execution engine units in the data processing system can be gradually reduced until they are completely decommissioned, enabling gray-scale updates of execution engine units.

[0142] In some embodiments, for the same type of task nodes, the newly added first type of execution engine unit has higher processing efficiency than the existing other types of execution engine units, resulting in higher processing efficiency of the data processing system. In one embodiment, the first type of execution engine unit is based on a first compiled language, while other types of execution engine units are based on different compiled languages. The first compiled language used by the first type of execution engine unit makes its processing efficiency higher. For example, the first type of execution engine unit is written in C++, and the second type of execution engine unit is written in Java.

[0143] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0144] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components illustrated as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0145] Some embodiments of this application also provide an electronic device corresponding to the data processing method provided in the foregoing embodiments, for executing the above-described data processing method.

[0146] Figure 6 The present invention illustrates a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device includes a communication interface 601, a processor 602, a memory 603, and a bus 604. The communication interface 601, processor 602, and memory 603 communicate with each other via the bus 604. The processor 602 can execute the data processing method described above by reading and executing machine-executable instructions corresponding to the control logic of the data processing method stored in the memory 603. The specific content of this method is described in the above embodiment and will not be repeated here.

[0147] The memory 603 mentioned in this embodiment can be any electronic, magnetic, optical, or other physical storage device, and can contain stored information such as executable instructions, data, etc. Specifically, the memory 603 can be RAM (Random Access Memory), flash memory, storage drive (such as hard disk drive), any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof. Communication between this system network element and at least one other network element is achieved through at least one communication interface 601 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc., can be used.

[0148] Bus 604 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 603 is used to store programs, and the processor 602 executes the programs after receiving execution instructions.

[0149] Processor 602 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 602 or by instructions in software form. The processor 602 can be a general-purpose processor, including a network processor (NP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware controls, etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor.

[0150] The electronic devices and data processing methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0151] This application also provides a computer-readable storage medium corresponding to the data processing method provided in the foregoing embodiments, which stores a computer program (i.e., a program product) thereon. When the computer program is run by a processor, it executes the data processing method provided in any of the foregoing embodiments.

[0152] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0153] The computer-readable storage medium provided in the above embodiments of this application and the data processing method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0154] This application also provides a computer program product corresponding to the data processing method provided in the foregoing embodiments. The computer program product includes a computer program that is executed by a processor to implement the data processing method provided in the foregoing embodiments.

[0155] The computer program products provided in the above embodiments of this application and the data processing methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0156] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0157] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Receive data processing tasks; The data processing task is divided into multiple sub-tasks; Based on the constraints of preset limiting rules, execution engine units are assigned to each subtask among multiple types of execution engine units; the multiple types of execution engine units include a first type of execution engine unit; the preset limiting rules are used to limit whether the first type of execution engine unit supports processing task nodes in each subtask; the task nodes include abstract syntax tree nodes or function implementation nodes; Each subtask is distributed to the assigned execution engine unit, so that the corresponding execution engine unit executes the assigned subtask.

2. The data processing method according to claim 1, characterized in that, The constraint based on preset limiting rules, which assigns execution engine units to each subtask among multiple types of execution engine units, includes: Based on the constraints of preset limiting rules, it is determined whether the first type of execution engine unit supports processing each task node in the first subtask; the first subtask is any one of the multiple subtasks. If the first type of execution engine unit does not support processing any task node in the first subtask, the first subtask is assigned to an execution engine unit other than the first type of execution engine unit among the multiple types of execution engine units; If the first type of execution engine unit supports processing each task node in the first subtask, the first subtask is assigned to the first type of execution engine unit.

3. The data processing method according to claim 2, characterized in that, The determination of whether the first type of execution engine unit supports processing each task node in the first subtask based on the constraints of preset limiting rules includes: Based on the target attributes of each task node in the first subtask, determine whether the first type of execution engine unit supports processing each task node in the first subtask; wherein, the target attributes of each task node are pre-configured target attributes of the task node type to which each task node belongs, and the target attributes are used to indicate the preset limiting rules.

4. The data processing method according to any one of claims 1-3, characterized in that, For the same type of task node, the first type of execution engine unit has a higher processing efficiency than other types of execution engine units, where other types of execution engine units are execution engine units other than the first type of execution engine unit among the multiple types of execution engine units.

5. The data processing method according to any one of claims 1-3, characterized in that, For the second subtask among the plurality of subtasks, the step of distributing each subtask to the assigned execution engine unit includes: Based on a pre-configured target protocol, the second subtask is converted into second subtask data; the second subtask is a subtask among the plurality of subtasks to be distributed to the first execution engine unit; the target protocol is used to enable the first execution engine node to parse the second subtask data to obtain the second subtask; the first execution engine unit is a first type of execution engine unit allocated for the second subtask. The second subtask data is distributed to the first execution engine unit.

6. The data processing method according to any one of claims 1-3, characterized in that, The constraint based on preset limiting rules, which assigns execution engine units to each subtask among multiple types of execution engine units, includes: Based on the running status information of the first type of execution engine unit, determine whether to suspend the use of the first type of execution engine unit; Based on the constraints of the preset limiting rules and the judgment result of whether to suspend the use of the first type of execution engine unit, an execution engine unit is assigned to each subtask among the multiple types of execution engine units.

7. A data processing system, characterized in that, The system includes multiple types of execution engine units and task coordinators; The multiple types of execution engine units include the first type of execution engine unit; The task coordinator is used to receive data processing tasks; The data processing task is divided into multiple sub-tasks; Based on the constraints of preset limiting rules, execution engine units are assigned to each subtask among the multiple types of execution engine units; the preset limiting rules are used to limit whether the first type of execution engine unit supports processing the task nodes in each subtask; The task nodes include abstract syntax tree nodes or function implementation nodes; Each subtask is distributed to the assigned execution engine unit, so that the corresponding execution engine unit executes the assigned subtask.

8. The data processing system according to claim 7, characterized in that, The multiple types of execution engine units also include a second type of execution engine unit. The first type of execution engine unit is a new type of execution engine unit added to the data processing system relative to the second type of execution engine unit.

9. The data processing system according to claim 7 or 8, characterized in that, The task coordinator is configured with a first protocol and / or a second protocol; The first protocol is used to define the conversion rules for the task coordinator to convert the first subtask into first subtask data; the first subtask is the subtask among the plurality of subtasks to be distributed to the first execution engine unit; The first protocol is used to enable the first execution engine unit to parse the first subtask data and obtain the first subtask; The first execution engine unit is a first type of execution engine unit allocated for the first subtask; The second protocol enables the task coordinator to parse the runtime statistics sent by the first type of execution engine unit.

10. The data processing system according to claim 7 or 8, characterized in that, The first type of execution engine unit is configured with a third protocol and / or a fourth protocol; The third protocol is used to define the data format for the first type of execution engine unit to send heartbeat information to the task coordinator; The fourth protocol is used to enable the first type of execution engine unit to parse the data sent by the third type of execution engine unit, or to define the data format sent to the third type of execution engine unit; The third type of execution engine unit is any type of execution engine unit other than the first type of execution engine unit among the multiple types of execution engine units.

11. A distributed online analytical processing system, characterized in that, Includes various execution engine units and task coordinators; The multiple types of execution engine units include a first type of execution engine unit, which is a newly added type of execution engine unit in the distributed online analytical processing system. The task coordinator is used to receive data processing tasks sent by the client for the data source and parse the data processing tasks into multiple sub-tasks. Based on the constraints of preset limiting rules, an execution engine unit is assigned to each subtask among the multiple types of execution engine units. The preset limiting rules are used to limit whether the first type of execution engine unit supports processing the task nodes in each subtask. The task nodes include abstract syntax tree nodes or function implementation nodes; Each subtask is distributed to an assigned execution engine unit; the execution engine unit assigned to each subtask in the data processing task is used to receive and execute the subtask sent by the task coordinator, and to collaboratively implement the data processing task, including feeding back the processing results to the client based on the processing of data in the data source.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the program to implement the method as described in any one of claims 1-7.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-7.

14. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 1-7.

Images