Task processing device and task processing method

Abstract

The application relates to a task processing device and a task processing method. The task processing device comprises a scheduling system configured to: display a task configuration page; in response to a user input operation on the task configuration page, acquire a plurality of configuration information corresponding to a plurality of task nodes, the configuration information comprising a task type, a task dependency relationship and a task execution parameter of the corresponding task node; analyze the plurality of configuration information to generate a directed acyclic graph (DAG) topology, the DAG topology comprising the plurality of task nodes and an execution sequence of the plurality of task nodes; based on the DAG topology, generate a DAG instance; and a multi-heterogeneous execution engine configured to execute the DAG instance in response to a scheduling instruction of the scheduling system.

Description

Technical Field

[0001] This application relates to data processing technology. More specifically, it relates to a task processing device and a task processing method. Background Technology

[0002] In the field of distributed data synchronization, the existing technical architecture has systemic defects that severely restrict the efficiency and reliability of large-scale data processing. Specifically, each execution engine (Python / Perl scripts, Kettle / Informatica and other ETL tools) needs to be deployed and scheduled independently, which results in task status being scattered across multiple consoles. The lack of unified monitoring leads to an increase in operation and maintenance costs of more than 30%. Moreover, cross-platform task dependencies need to be configured manually, and complex workflow orchestration takes more than 2 hours, resulting in low task execution efficiency. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, embodiments of this application provide a task processing device and a task processing method.

[0004] In a first aspect, embodiments of this application provide a task processing device, including: a scheduling system configured to: display a task configuration page; in response to a user's input operation on the task configuration page, obtain multiple configuration information corresponding to multiple task nodes, the configuration information including the task type, task dependency relationship and task execution parameters of the corresponding task node; parse the multiple configuration information to generate a directed acyclic graph (DAG) topology, the DAG topology including the multiple task nodes and the execution order of the multiple task nodes; generate a DAG instance based on the DAG topology; and multiple heterogeneous execution engines configured to: execute the DAG instance in response to scheduling instructions from the scheduling system.

[0005] In this embodiment, the scheduling system sets up a task configuration page. Users can input information on this page to configure multiple task nodes, including task type, dependencies, and execution parameters. The system then parses this configuration information to dynamically generate a Directed Acyclic Graph (DAG) topology (including the task nodes and their execution order). Based on this DAG topology, DAG instances are dynamically generated and executed. These instances are then executed via multiple heterogeneous execution engines responding to scheduling instructions from the scheduling system. This unified scheduling system and multi-heterogeneous execution engine architecture enable the dynamic generation of DAG topologies and instances based on user-inputted configuration information for multiple task nodes. This eliminates the need for manual configuration of DAG topologies and instances, improving generation efficiency. Furthermore, the unified scheduling system schedules these instances to multiple heterogeneous execution engines, achieving unified scheduling of tasks across platforms and supporting execution of task instances within DAG instances by multiple heterogeneous execution engines, thus improving task execution efficiency.

[0006] In some embodiments of this application, the scheduling system is specifically configured to: determine the local operator corresponding to the target task type included in the target configuration information as the target operator, and the target configuration information is any one of the plurality of configuration information; based on the target operator, encapsulate the task execution parameters included in the target configuration information to obtain the target task node corresponding to the target configuration information, so as to obtain the plurality of task nodes based on the plurality of configuration information respectively; determine the execution order of the plurality of task nodes based on the task dependencies included in the plurality of configuration information respectively; and determine the DAG topology based on the plurality of task nodes and the execution order of the plurality of task nodes.

[0007] In this embodiment, by setting all task nodes to be encapsulated by the local operator built into the task processing device itself, without requiring the user to pre-compile the encapsulator, DAG topology can be dynamically generated based on the task configuration information of multiple task nodes.

[0008] In some embodiments of this application, the scheduling system is specifically configured to: when the target task type is a first command script type, determine the first local operator corresponding to the first command script type as the target operator, and different command script types correspond to different operators; when the target task type is a first ETL type, determine the second local operator corresponding to the first ETL type as the target operator, and different ETL types correspond to different operators.

[0009] In this embodiment, the specific task type to which the target task type belongs determines the corresponding local operator. In this way, a suitable local operator can be accurately determined, thereby enabling dynamic and accurate generation of DAG topology.

[0010] In some embodiments of this application, when the task type included in the configuration information is a command script type, the task execution parameters corresponding to the configuration information include command script parameters; when the task type included in the configuration information is an ETL type, the task execution parameters corresponding to the configuration information include data source parameters and data destination parameters of the corresponding task node.

[0011] In this embodiment, the task execution parameters can be different for task nodes of different task types, so that the corresponding DAG topology can be accurately generated based on the task execution parameters corresponding to different task types.

[0012] In some embodiments of this application, the task processing device further includes a lineage resolution system, configured to: when the configuration information also includes lineage information of the corresponding task nodes, obtain full-link lineage information corresponding to the multiple task nodes based on the multiple configuration information; when the configuration information does not include the lineage information of the corresponding task nodes, obtain full-link lineage information corresponding to the multiple task nodes by parsing the task execution logic corresponding to each task node during the execution of the DAG instance; wherein, the full-link lineage information includes the intra-node lineage relationship of each task node and the data flow lineage relationship between the nodes corresponding to the multiple task nodes.

[0013] In this embodiment of the application, under the architecture of a unified scheduling system and multiple heterogeneous execution engines, end-to-end lineage tracing can be achieved, providing end-to-end observability for data governance.

[0014] In some embodiments of this application, the lineage resolution system is specifically configured to: when the target task node is a command script type task, obtain the target lineage information of the target task node by parsing the target command script of the target task node. The target lineage information includes the intra-node lineage relationship of the target task node and the data flow lineage relationship between the nodes corresponding to the target task node; when the target task node is an ETL type task, obtain the intra-node lineage relationship of the target task node and the data flow lineage relationship between the nodes corresponding to the target task node by parsing the target SQL statement of the target task node.

[0015] In this embodiment of the application, during the execution of a task instance, for a command script type task, the lineage information of the corresponding task is obtained by parsing the corresponding command script; for an RTL type task, the lineage information of the corresponding task is obtained by parsing the corresponding SQL statement and extracting the mapping relationship. In this way, for different types of tasks, the full-link lineage information of the multiple task nodes can be obtained by parsing the task execution logic of each task node.

[0016] In some embodiments of this application, the lineage resolution system is specifically configured to: when the target task node is a command script type task, obtain the target execution command corresponding to the target task node; if the target execution command matches a first preset execution command among multiple preset execution commands, determine the lineage information corresponding to the first preset execution command as the target lineage information; if the target execution command does not match any of the multiple preset execution commands, perform preliminary parsing on the target execution command to obtain script-related information of the target command script; based on the script-related information, obtain the target command script from the file system; parse the target command script to obtain a script parsing result; and determine the target lineage information based on the script parsing result.

[0017] In this embodiment of the application, when the task node is a command script type task, the lineage information is determined by parsing the execution command corresponding to the task node, and then the full-link lineage information corresponding to the multiple task nodes is determined.

[0018] In some embodiments of this application, the scheduling system is further configured to: before executing the DAG instance, obtain multiple load information corresponding to multiple service nodes for the target task instance in the DAG instance, the load information including the computing load utilization, data load throughput and data transmission latency of the corresponding service node; determine the comprehensive score corresponding to each service node based on the multiple load information; and distribute the target task instance to the target service node with the highest comprehensive score, so that the corresponding heterogeneous execution engine on the target service node executes the target task instance.

[0019] In this embodiment, by obtaining multiple load information corresponding to multiple service nodes, calculating the comprehensive score corresponding to each service node based on the multiple load information, and then distributing the task instance to the target service node with the highest comprehensive score, the scheduling system can achieve optimal utilization of cluster resources.

[0020] In some embodiments of this application, the scheduling system is specifically configured to: determine the corresponding target comprehensive score formula based on the task type of the target task instance, with different task types corresponding to different comprehensive score formulas; and determine the comprehensive score corresponding to each service node based on the multiple load information and the target comprehensive score formula.

[0021] In this embodiment of the application, by setting different comprehensive score formulas corresponding to different task types, the optimal service node for executing the task corresponding to the current task type can be determined based on multiple load information corresponding to each service node and the comprehensive score formula of the corresponding task type, which is beneficial to improving the execution efficiency of the task.

[0022] Secondly, embodiments of this application provide a task processing method applied to a task processing device, comprising: displaying a task configuration page through a scheduling system; in response to a user's input operation on the task configuration page, obtaining multiple configuration information corresponding to multiple task nodes, the configuration information including the task type, task dependency relationship and task execution parameters of the corresponding task node; parsing the multiple configuration information to generate a directed acyclic graph (DAG) topology, the DAG topology including the multiple task nodes and the execution order of the multiple task nodes; generating a DAG instance based on the DAG topology; and executing the DAG instance through multiple heterogeneous execution engines in response to scheduling instructions from the scheduling system.

[0023] Thirdly, embodiments of this application provide a computer-readable storage medium, including: storing a computer program on the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the task processing method as shown in the second aspect.

[0024] Fourthly, embodiments of this application provide a computer program product, including: when the computer program product is run on a computer, causing the computer to implement the task processing method as shown in the second aspect. Attached Figure Description

[0025] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0026] Figure 1 A schematic diagram of the structure of a task processing device 10 according to some embodiments is shown; Figure 2 A schematic diagram of the structure of a scheduling system 11 according to some embodiments is shown; Figure 3A flowchart corresponding to the scheduling system 11 according to some embodiments is shown; Figure 4 A schematic diagram of the structure of a bloodline analysis system 13 according to some embodiments is shown; Figure 5 A schematic flowchart corresponding to the bloodline analysis system 13 according to some embodiments is shown; Figure 6 One of the flowcharts of a task processing method according to some embodiments is shown; Figure 7 A second flowchart illustrating a task processing method according to some embodiments is shown; Figure 8 A third flowchart illustrating a task processing method according to some embodiments is shown; Figure 9 A fourth flowchart illustrating a task processing method according to some embodiments is shown; Figure 10 Fifth of a series of flowcharts illustrating a task processing method according to some embodiments is shown. Detailed Implementation

[0027] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0028] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0029] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0030] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0031] In some embodiments of this application, the task processing device may be a server device, a server cluster, a data platform composed of multiple server devices or multiple server clusters, a task execution engine in the data platform, or other feasible devices. The specific device can be determined according to the actual situation and is not limited here.

[0032] This application's embodiments belong to the field of distributed computing and data integration technology, and are particularly suitable for large-scale data migration scenarios in fields such as finance and telecommunications. Traditional solutions suffer from the following three core problems: the difficulty of adapting to heterogeneous environments, as traditional systems require separate deployment of scheduling modules for different execution engines, leading to architectural redundancy; uneven resource utilization, as static server binding mode causes both hot node overload and resource idleness; and insufficient operational observability, as the task execution process lacks a unified monitoring system, making it difficult to trace data lineage.

[0033] In some embodiments of this application, the scheduling system is configured to: display a task configuration page; in response to user input on the task configuration page, obtain multiple configuration information corresponding to multiple task nodes, the configuration information including the task type, task dependency relationship and task execution parameters of the corresponding task node; parse the multiple configuration information to generate a directed acyclic graph (DAG) topology, the DAG topology including the multiple task nodes and the execution order of the multiple task nodes; generate a DAG instance based on the DAG topology; and execute the DAG instance.

[0034] The input operations can include text input on the task configuration page, selection of operation options, and uploading of script code, etc. The specific operations can be determined according to the actual situation and are not limited here.

[0035] In some embodiments of this application, the input operation may include input operations on the task type, task dependency relationship, and task execution parameters of the task node, and may also include input operations on other information, which are not limited here.

[0036] The task type can include command script type, ETL (Extract, Transform, Load) type, or other types, which are not limited here.

[0037] The command script type can be further subdivided into various dimensions of command script types, which can be determined according to the actual situation and are not limited here.

[0038] In this system, multiple task nodes correspond one-to-one with multiple configuration information, meaning that one task node corresponds to one configuration information.

[0039] The task configuration page may include at least one information configuration area corresponding to at least one task node, and may also include additional options for the information configuration area corresponding to the task node. The specifics can be determined according to the actual situation and are not limited here.

[0040] The information configuration area is used to configure the configuration information of the corresponding task nodes. The information configuration area includes required configuration items and optional configuration items, which can be determined according to the actual situation and are not limited here.

[0041] The task configuration page allows you to configure any of the following: task node identifier, task node description information, task node manager information, data source and destination information, host connection information, and command script information; other information can also be configured, which is not limited here.

[0042] In cases where the task execution parameters include command script information, the configuration area may also include a preview option for the input command script.

[0043] Among them, task execution parameters are all the core parameters required to complete a specific data processing task (command script type / ETL type). They are the "business soul" of the task node, determining what the task node should execute, how it should execute, and which engine should be used. The task execution parameters can be determined based on the information entered by the user on the task configuration page, and can vary depending on the task type; no specific limitations are set here.

[0044] In some embodiments of this application, a DAG topology can be obtained by dynamically orchestrating multiple configuration information using a dynamic process orchestration engine. The dynamic process engine may include a configuration parser, which parses multiple configuration information to dynamically generate the DAG topology.

[0045] It is understood that in the embodiments of this application, a mapping relationship from multiple configuration information to DAG topology is set (the number of multiple configuration information is not limited). Specifically, this includes how to map the configuration information of a task node to that task node, and how to map the configuration information of multiple task nodes to the execution order of those multiple task nodes.

[0046] In some embodiments of this application, a complete DAG topology structure containing task nodes, dependencies, and execution attributes is constructed in real time at runtime through rule-driven, metadata linkage, and event triggering methods.

[0047] In this embodiment of the application, a directed acyclic graph (DAG) dynamic generation technology based on pure configuration declarations (i.e. configuration information) is provided. Users only need to set the configuration information corresponding to the task nodes, and the system can directly and in real time convert multiple configuration information corresponding to multiple task nodes into executable workflow instances.

[0048] In this embodiment of the application, unlike traditional technologies where DAG topology generation requires a predefined DAG file as a template, this embodiment does not require any predefined DAG template. The structure of the DAG topology (task nodes and their dependencies) is completely parsed and dynamically constructed in real time by the user-submitted configuration.

[0049] In this embodiment, task parameters are not simply "injected" into a fixed template. Instead, configuration information serves as the basic input for constructing the DAG topology and defining task behavior, achieving a seamless, native transformation from configuration to an executable DAG and enabling deep integration of dynamic parameters. The structure of the DAG topology (which task nodes are present and how they are connected) is dynamically generated entirely from the task types, task dependencies, and task execution parameters of the task nodes in the configuration information.

[0050] In this embodiment, based on the user-submitted pure configuration declaration (including task type, dependencies, and task execution parameters), the system core engine (configuration parser + DAG generator) directly and in real time constructs a complete and executable DAG object in memory.

[0051] The mapping relationship from the configuration information of a task node to that task node can include the mapping from the configuration information to the corresponding information of the task node, as well as the default mapping of the information of task nodes not included in the configuration information (which can be empty or a default value, and is not limited here).

[0052] The execution order of task nodes can be a serial workflow (i.e., serial execution order), a parallel workflow (i.e., parallel execution order), or a mixed serial and parallel workflow (i.e., mixed serial and parallel execution order). The specific order can be determined according to the actual situation and is not limited here.

[0053] In some embodiments of this application, DAG instances can be dynamically generated based on the DAG topology using a DAG generator.

[0054] It is understood that in this embodiment of the application, a mapping relationship from DAG topology to DAG instance is set, which can be determined according to the actual situation and is not limited here.

[0055] Among them, multi-heterogeneous execution engines refer to a distributed execution capability cluster composed of execution units with different technical systems, execution protocols, and functional positioning, which can be uniformly scheduled.

[0056] A DAG instance can include multiple task instances, and multiple task instances can correspond to multiple task nodes.

[0057] It is understandable that the scheduling system can decompose the task instances included in the DAG instance, and then distribute the decomposed task instances to the corresponding heterogeneous execution engines for execution. Multiple heterogeneous execution engines can execute the task instances distributed by the scheduling system, and different types of heterogeneous execution engines can execute different types of scheduled tasks distributed by the scheduling system.

[0058] In this embodiment, the scheduling system sets up a task configuration page. Users can input information on this page to configure multiple task nodes, including task type, dependencies, and execution parameters. The system then parses this configuration information to dynamically generate a Directed Acyclic Graph (DAG) topology (including the task nodes and their execution order). Based on this DAG topology, DAG instances are dynamically generated and executed. These instances are then executed via multiple heterogeneous execution engines responding to scheduling instructions from the scheduling system. This unified scheduling system and multi-heterogeneous execution engine architecture enable the dynamic generation of DAG topologies and instances based on user-inputted configuration information for multiple task nodes. This eliminates the need for manual configuration of DAG topologies and instances, improving generation efficiency. Furthermore, the unified scheduling system schedules these instances to multiple heterogeneous execution engines, achieving unified scheduling of tasks across platforms and supporting execution of task instances within DAG instances by multiple heterogeneous execution engines, thus improving task execution efficiency.

[0059] In some embodiments of this application, the scheduling system is specifically configured to: determine the local operator corresponding to the target task type included in the target configuration information as the target operator, and the target configuration information is any one of the plurality of configuration information; based on the target operator, encapsulate the task execution parameters included in the target configuration information to obtain the target task node corresponding to the target configuration information, so as to obtain the plurality of task nodes based on the plurality of configuration information respectively; determine the execution order of the plurality of task nodes based on the task dependencies included in the plurality of configuration information respectively; and determine the DAG topology based on the plurality of task nodes and the execution order of the plurality of task nodes.

[0060] The local operator can be an operator from the task processing device's local operator library. The operator can be an operator or a specific function; there are no restrictions here.

[0061] It can be understood that the local operator is a standardized encapsulation carrier local to the task processing device, and the task execution parameters are the business core and execution basis of the task node. By encapsulating the task execution parameters through the local operator, the programmable and schedulable task nodes in the DAG topology are finally formed.

[0062] In this embodiment of the application, unlike traditional technologies where workflow engines must build task nodes based on user-precompiled Operators or specific function templates, the solution to be protected in this embodiment does not rely on any user-precompiled Operators or functions. The execution logic of the task node is dynamically determined by the task type and task execution parameters defined in the configuration.

[0063] In this embodiment, by setting all task nodes to be encapsulated by the local operator built into the task processing device itself, without requiring the user to pre-compile the encapsulator, DAG topology can be dynamically generated based on the task configuration information of multiple task nodes.

[0064] In some embodiments of this application, the scheduling system is specifically configured to: when the target task type is a first command script type, determine the first local operator corresponding to the first command script type as the target operator, and different command script types correspond to different operators; when the target task type is a first ETL type, determine the second local operator corresponding to the first ETL type as the target operator, and different ETL types correspond to different operators.

[0065] The first command script type can be one of multiple command script types, and these multiple command script types correspond to multiple local operators, which is not limited here.

[0066] The first ETL type can be one of multiple ETL types, and the multiple ETL types correspond to multiple local operators, which is not limited here.

[0067] In this embodiment, the corresponding local operator is determined by the specific task type to which the target task type belongs. In this way, a suitable local operator can be accurately determined, thereby enabling dynamic and accurate generation of DAG topology.

[0068] In some embodiments of this application, when the task type included in the configuration information is a command script type, the task execution parameters corresponding to the configuration information include command script parameters; when the task type included in the configuration information is an ETL type, the task execution parameters corresponding to the configuration information include data source parameters and data destination parameters of the corresponding task node.

[0069] The specific parameters of the command script can include the script path, command, configuration file path, etc., which are not limited here.

[0070] In the case of a task type of command script, this configuration information may also include other information, which is not limited here.

[0071] The data source parameters and data destination parameters may include the data source type, data source path, and the path of the database table to which the data source belongs, etc., which are not limited here.

[0072] In the case of ETL task type, this configuration information may also include other information (such as field mapping parameters), which is not limited here.

[0073] In this embodiment, the task execution parameters can be different for task nodes of different task types, so that the corresponding DAG topology can be accurately generated based on the task execution parameters corresponding to different task types.

[0074] In some embodiments of this application, the task processing device further includes a lineage resolution system, configured to: when the configuration information also includes lineage information of the corresponding task nodes, obtain full-link lineage information corresponding to the multiple task nodes based on the multiple configuration information; when the configuration information does not include the lineage information of the corresponding task nodes, obtain full-link lineage information corresponding to the multiple task nodes by parsing the task execution logic corresponding to each task node during the execution of the DAG instance; wherein, the full-link lineage information includes the intra-node lineage relationship of each task node and the data flow lineage relationship between the nodes corresponding to the multiple task nodes.

[0075] This includes using metadata auto-association technology to extract the data table structures, field mapping relationships, and transformation logic between the source and target ends during the task parsing phase. A versioned lineage graph can be constructed to record the history of table structure changes and the evolution of data flow, supporting lineage tracing back to any point in time.

[0076] It's understandable that if the user has already filled in the lineage information for each task node on the task configuration page, then the full-link lineage information for multiple task nodes can be determined based on the user-filled lineage information and the task dependencies of each node. If the user hasn't filled in the lineage information for each task node on the task configuration page, then during the execution of the DAG instance, the full-link lineage information for these multiple task nodes can be obtained by parsing the task execution logic corresponding to each task node. Thus, under a unified scheduling system and a multi-heterogeneous execution engine architecture, full-link lineage tracing can be achieved without manual configuration of lineage relationships, providing end-to-end observability for data governance.

[0077] In some embodiments of this application, the lineage resolution system is specifically configured to: when the target task node is a command script type task, obtain the target lineage information of the target task node by parsing the target command script of the target task node. The target lineage information includes the intra-node lineage relationship of the target task node and the data flow lineage relationship between the nodes corresponding to the target task node; when the target task node is an ETL type task, obtain the intra-node lineage relationship of the target task node and the data flow lineage relationship between the nodes corresponding to the target task node by parsing the target SQL statement of the target task node.

[0078] In this embodiment of the application, during the execution of a task instance, for a command script type task, the lineage information of the corresponding task is obtained by parsing the corresponding command script; for an RTL type task, the lineage information of the corresponding task is obtained by parsing the corresponding SQL statement and extracting the mapping relationship. In this way, for different types of tasks, the full-link lineage information of the multiple task nodes can be obtained by parsing the task execution logic of each task node.

[0079] In some embodiments of this application, the lineage resolution system is specifically configured to: when the target task node is a command script type task, obtain the target execution command corresponding to the target task node; if the target execution command matches a first preset execution command among multiple preset execution commands, determine the lineage information corresponding to the first preset execution command as the target lineage information; if the target execution command does not match any of the multiple preset execution commands, perform preliminary parsing on the target execution command to obtain script-related information of the target command script; based on the script-related information, obtain the target command script from the file system; parse the target command script to obtain a script parsing result; and determine the target lineage information based on the script parsing result.

[0080] The target execution command is the execution command corresponding to the target task instance of the target task node in the DAG instance.

[0081] In this embodiment, multiple preset execution commands are stored, along with their corresponding lineage information. Different preset execution commands correspond to different lineage information.

[0082] The file system stores multiple command scripts, each corresponding to different script-related information.

[0083] The script-related information can include script identifiers, script indexes, etc., which are not limited here.

[0084] For example, such as Figure 1As shown, the task processing device 10 provided in this embodiment includes a scheduling system 11, a multi-heterogeneous execution engine 12, and a lineage analysis system 13. The multi-heterogeneous execution engine 12 includes various heterogeneous execution engines, which are not limited here.

[0085] For example, combined Figure 1 ,like Figure 2 As shown, the scheduling system 11 may include a configuration center 111, a configuration parser 112, a DAG generator 113, and a task scheduler 114. The configuration center 111 is used to display the task configuration page and obtain multiple configuration information; the configuration parser 112 is used to generate a DAG topology based on the multiple configuration information; the DAG generator 113 is used to generate DAG instances based on the DAG topology; and the task scheduler 114 is used to decompose the DAG instance to obtain multiple task instances and distribute the multiple task instances to multiple heterogeneous execution engines. Specifically, as... Figure 3 As shown, 301. Configuration center 111 obtains multiple configuration information based on user input; 302. Configuration center 111 sends the multiple configuration information to configuration parser 112; 303. Configuration parser 112 parses the multiple configuration information to dynamically construct multiple task nodes and determines the execution order of the multiple task nodes; 304. Configuration parser 112 dynamically constructs a DAG topology based on the multiple task nodes and their execution order; 305. DAG generator 113 dynamically generates a DAG instance based on the DAG topology; 306. DAG generator 113 sends the DAG instance to task scheduler 114; 307. Task scheduler 114 schedules multiple heterogeneous execution engines to execute the DAG instance.

[0086] For example, combined Figure 1 and Figure 2 ,like Figure 4 As shown, the lineage analysis system 13 may include a metadata crawler 131, an AI parsing engine 132, a file system 133, and a lineage information database 134. The metadata crawler 131 is used to collect various descriptive, configurable, and runtime metadata. After cleaning, standardization, and association, it is uniformly integrated into the enterprise metadata center, providing underlying data support for end-to-end control and data governance. The AI ​​parsing engine 132 is used to parse execution commands and command scripts. The file system 133 stores command scripts. The lineage information database 134 stores the correspondence between execution commands and lineage information. Specifically, as... Figure 5As shown, 501, when the target task node is a command script type task, the task scheduler 114 triggers the target execution command; 502, the metadata crawler 131 matches the target execution command with multiple preset execution commands; 503, when the target execution command matches the first preset execution command among the multiple preset execution commands, the metadata crawler 131 retrieves the lineage information corresponding to the first preset execution command from the lineage information database 134; 504, the metadata crawler 131 determines the lineage information corresponding to the first preset execution command as the target lineage information; 505, when the target execution command does not match any of the multiple preset execution commands, the metadata crawler 131 sends the target execution command to the AI ​​parsing engine 132; 506 507. The AI ​​parsing engine 132 performs preliminary parsing of the target execution command to obtain script-related information of the target command script; 508. The AI ​​parsing engine 132 returns the script-related information to the metadata crawler 131; 509. Based on the script-related information, the metadata crawler 131 retrieves the target command script from the file system 133; 510. The metadata crawler 131 sends the target command script to the AI ​​parsing engine 132; 511. The AI ​​parsing engine 132 parses the target command script to obtain the script parsing result; 512. The AI ​​parsing engine 132 returns the script parsing result to the metadata crawler 131; 513. Based on the script parsing result, the metadata crawler 131 retrieves the target lineage information from the lineage information database 134.

[0087] In this embodiment of the application, when the task node is a command script type task, the lineage information is determined by parsing the execution command corresponding to the task node, and then the full-link lineage information corresponding to the multiple task nodes is determined.

[0088] In some embodiments of this application, for ETL type tasks, lineage information can be determined based on the lineage information filled in or the data source and destination in the configuration information filled in.

[0089] In some embodiments of this application, the scheduling system is further configured to: before executing the DAG instance, obtain multiple load information corresponding to multiple service nodes for the target task instance in the DAG instance, the load information including the computing load utilization, data load throughput and data transmission latency of the corresponding service node; determine the comprehensive score corresponding to each service node based on the multiple load information; and distribute the target task instance to the target service node with the highest comprehensive score, so that the corresponding heterogeneous execution engine on the target service node executes the target task instance.

[0090] Among them, heterogeneous execution engines are software-level execution capability units. They are dedicated execution components with different technology stacks and responsible for executing different types of tasks. "Heterogeneous" refers to the fact that their technology systems, execution protocols, and task types are different. Heterogeneous execution engines are divided into command script execution engines, ETL execution engines, and general-purpose execution engines. Each type of execution engine can be further subdivided into several subtypes of execution engines, which are not limited here.

[0091] Service nodes are runtime units at the resource level. They are node units in a distributed architecture that provide computing, storage, and network resources for task execution. They are divided into two categories: physical / virtual service nodes or logical service nodes.

[0092] In this context, the service node and the heterogeneous execution engine have a carrier-support relationship. The service node is the deployment carrier and operating environment for the heterogeneous execution engine. All heterogeneous execution engines must be installed and deployed on the service node in order to be started, receive instructions, and execute tasks. The heterogeneous execution engine is a capability component on the service node. Only when the service node has deployed the corresponding heterogeneous execution engine can it have the ability to execute a certain type of task.

[0093] In a distributed architecture, service nodes and heterogeneous execution engines are not bound one-to-one, but rather deployed flexibly in a many-to-many manner, supporting high availability, load balancing, and elastic scaling. A service node can deploy multiple heterogeneous execution engines to become a comprehensive execution node capable of handling various types of tasks; a heterogeneous execution engine can be deployed on multiple service nodes, and the same type of heterogeneous execution engine can be deployed on multiple service nodes to form an engine cluster, enabling parallel task execution and failover.

[0094] In this system, the unified scheduling system acts as the central hub. It does not directly interact with the heterogeneous execution engines, but instead first schedules task instances to service nodes, where the heterogeneous execution engines execute them, forming a complete chain. For example, the scheduling system generates a DAG instance, which is then broken down into task instances. Based on load, engine type, and resource availability, the scheduling system allocates the task instances to target service nodes. The target service node receives the instruction and wakes up the corresponding heterogeneous execution engine deployed on its local machine. The heterogeneous execution engine loads the task instance, completes the actual execution, and sends the result back to the service node, which then reports it to the scheduling system.

[0095] Among them, computing load utilization can be CPU / GPU utilization, data load throughput can be I / O throughput, and data transmission latency can be node transmission latency.

[0096] The comprehensive score for the current service node can be obtained by weighting the following factors: the computing load utilization of the current service node, the ratio of the current service node's data throughput to the sum of the data throughput of all service nodes (denoted as the data throughput percentage), and the ratio of the current service node's data transmission latency to the sum of the data throughput of all service nodes (denoted as the data transmission latency percentage). The weights for computing load utilization, data throughput percentage, and data transmission latency percentage can be determined based on actual circumstances and are not limited here.

[0097] In this embodiment, by obtaining multiple load information corresponding to multiple service nodes, calculating the comprehensive score corresponding to each service node based on the multiple load information, and then distributing the task instance to the target service node with the highest comprehensive score, the scheduling system can achieve optimal utilization of cluster resources.

[0098] In some embodiments of this application, the scheduling system is specifically configured to: determine the corresponding target comprehensive score formula based on the task type of the target task instance, with different task types corresponding to different comprehensive score formulas; and determine the comprehensive score corresponding to each service node based on the multiple load information and the target comprehensive score formula.

[0099] The comprehensive score formula is a weighted formula that calculates the comprehensive score for the current service node by weighting its computational load utilization, data throughput ratio, and data transmission latency ratio. Different weights can be assigned to computational load utilization, data throughput ratio, and data transmission latency ratio based on the task type, resulting in different comprehensive score formulas for different task types.

[0100] In this embodiment of the application, by setting different comprehensive score formulas corresponding to different task types, the optimal service node for executing the task corresponding to the current task type can be determined based on multiple load information corresponding to each service node and the comprehensive score formula of the corresponding task type, which is beneficial to improving the execution efficiency of the task.

[0101] In some embodiments of this application, a cross-engine adaptation layer is designed, employing a remote command channel for script-based tasks such as Python / Perl, and calling the native data pipeline interface for ETL tools such as SeaTunnel; a dynamic parameter injection mechanism is developed to support real-time rendering of environment variables and custom parameters; a multi-dimensional resource evaluation model is constructed to calculate node health by comprehensively considering indicators such as CPU load, memory utilization, disk I / O throughput, and network bandwidth; a dynamic weight adjustment algorithm is designed to automatically optimize indicator weights according to task type (e.g., increasing CPU weight for computationally intensive tasks and emphasizing network indicators for data transmission tasks); and a fault self-healing mechanism is implemented, ensuring task continuity through heartbeat detection and rapid switching to backup nodes, supporting breakpoint resumption and automatic retries.

[0102] To illustrate this solution in more detail, the following will use examples to illustrate it. Figures 6 to 10 To explain, it is understandable that Figures 6 to 10 The steps involved may include more or fewer steps in actual implementation, and the order of these steps may also differ, as long as the task execution method provided in the embodiments of this application can be achieved. The executing entity of the task execution method can be a task processing device, or a functional module or functional entity within the task processing device that can implement the task execution method; no limitation is made here. Furthermore, the specific description of the task execution method provided in the embodiments of this application can be found in the relevant description of the task processing device described above, and the same or similar technical effects can be achieved; further details are omitted here.

[0103] Figure 6 The flowchart illustrates the steps of implementing a task execution method according to one or more embodiments of this application. The task execution method may include the following steps S601 to S605.

[0104] S601, Display the task configuration page through the scheduling system.

[0105] S602. The scheduling system responds to the user's input on the task configuration page and obtains multiple configuration information corresponding to multiple task nodes.

[0106] This configuration information includes the task type, task dependencies, and task execution parameters for the corresponding task node.

[0107] S603. The scheduling system parses the multiple configuration information to generate a directed acyclic graph (DAG) topology.

[0108] The DAG topology includes the multiple task nodes and their execution order.

[0109] S604. Generate a DAG instance based on the DAG topology through the scheduling system.

[0110] S605. Execute the DAG instance in response to the scheduling instructions of the scheduling system through multiple heterogeneous execution engines.

[0111] In some embodiments of this application, combined with Figure 6 ,like Figure 7 As shown, the above S603 can be specifically implemented by the following S603a to S603d.

[0112] S603a. The local operator corresponding to the target task type included in the target configuration information is determined as the target operator through the scheduling system.

[0113] The target configuration information can be any one of the multiple configuration information.

[0114] S603b: The scheduling system encapsulates the task execution parameters included in the target configuration information based on the target operator to obtain the target task node corresponding to the target configuration information, so as to obtain the multiple task nodes based on the multiple configuration information respectively.

[0115] S603c: The scheduling system determines the execution order of the multiple task nodes based on the task dependencies included in the multiple configuration information.

[0116] S603d. The scheduling system determines the DAG topology based on the multiple task nodes and their execution order.

[0117] In some embodiments of this application, combined with Figure 7 ,like Figure 8 As shown, the above S603a can be specifically implemented by the following S603a1 to S603a2.

[0118] S603a1. If the target task type is a first command script type, the first local operator corresponding to the first command script type is determined as the target operator.

[0119] Different command script types correspond to different operators.

[0120] S603a2, If the target task type is the first ETL type, the second local operator corresponding to the first ETL type is determined as the target operator.

[0121] Different ETL types correspond to different operators.

[0122] In some embodiments of this application, when the task type included in the configuration information is a command script type, the task execution parameters corresponding to the configuration information include command script parameters; when the task type included in the configuration information is an ETL type, the task execution parameters corresponding to the configuration information include data source parameters and data destination parameters of the corresponding task node.

[0123] In some embodiments of this application, combined with Figure 8 ,like Figure 9 As shown, the task processing method provided in this application embodiment may further include the following S606 to S607.

[0124] S606. If the configuration information also includes the lineage information of the corresponding task nodes, the lineage analysis system obtains the full-link lineage information of the multiple task nodes based on the multiple configuration information.

[0125] S607. If the configuration information does not include the lineage information of the corresponding task node, the lineage resolution system obtains the full-link lineage information of the multiple task nodes by parsing the task execution logic corresponding to each task node during the execution of the DAG instance.

[0126] The end-to-end lineage information includes the intra-node lineage relationships of each task node and the data flow lineage relationships between the nodes corresponding to these multiple task nodes.

[0127] In some embodiments of this application, combined with Figure 9 ,like Figure 10 As shown, the above S607 can be specifically implemented by the following S607a to S607b.

[0128] S607a. When the target task node is a command script type task, the target lineage information of the target task node is obtained by parsing the target command script of the target task node.

[0129] The target lineage information includes the intra-node lineage of the target task node, as well as the data flow lineage between the nodes corresponding to the target task node.

[0130] S607b: When the target task node is an ETL type task, the node lineage of the target task node and the data flow lineage between the nodes corresponding to the target task node are obtained by parsing the target SQL statement of the target task node.

[0131] In some embodiments of this application, the above-mentioned S607a can be implemented through the following process: when the target task node is a command script type task, obtain the target execution command corresponding to the target task node; when the target execution command matches the first preset execution command among multiple preset execution commands, determine the lineage information corresponding to the first preset execution command as the target lineage information; when the target execution command does not match any of the multiple preset execution commands, perform preliminary parsing on the target execution command to obtain script-related information of the target command script; based on the script-related information, obtain the target command script from the file system; parse the target command script to obtain the script parsing result; and determine the target lineage information based on the script parsing result.

[0132] In some embodiments of this application, the task processing method provided in this application may further include: before executing the DAG instance, obtaining multiple load information corresponding to multiple service nodes for the target task instance in the DAG instance through a scheduling system, the load information including the computing load utilization, data load throughput and data transmission latency of the corresponding service node; determining the comprehensive score corresponding to each service node based on the multiple load information through the scheduling system; and distributing the target task instance to the target service node with the highest comprehensive score through the scheduling system, so that the corresponding heterogeneous execution engine on the target service node executes the target task instance.

[0133] In some embodiments of this application, the above-mentioned method of determining the comprehensive score corresponding to each service node based on the multiple load information by the scheduling system specifically includes: determining the corresponding target comprehensive score formula based on the task type of the target task instance, with different task types corresponding to different comprehensive score formulas; and determining the comprehensive score corresponding to each service node based on the multiple load information and the target comprehensive score formula.

[0134] The present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described task execution method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0135] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0136] The present invention provides a computer program product, comprising: when the computer program product is run on a computer, causing the computer to perform the above-described task execution method.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0138] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A task processing device, characterized in that, include: The scheduling system is configured to display the task configuration page; In response to the user's input operation on the task configuration page, multiple configuration information corresponding to multiple task nodes are obtained. The configuration information includes the task type, task dependency relationship and task execution parameters of the corresponding task node. The multiple configuration information is parsed to generate a directed acyclic graph (DAG) topology, which includes the multiple task nodes and their execution order. Based on the DAG topology, generate a DAG instance; Multiple heterogeneous execution engines are configured to execute the DAG instance in response to scheduling instructions from the scheduling system.

2. The task processing device according to claim 1, characterized in that, The scheduling system is specifically configured as follows: The local operator corresponding to the target task type included in the target configuration information is determined as the target operator, and the target configuration information is any one of the plurality of configuration information; Based on the target operator, the task execution parameters included in the target configuration information are encapsulated to obtain the target task node corresponding to the target configuration information, so as to obtain the multiple task nodes based on the multiple configuration information respectively; Based on the task dependencies included in the multiple configuration information, the execution order of the multiple task nodes is determined; The DAG topology is determined based on the plurality of task nodes and the execution order of the plurality of task nodes.

3. The task processing device according to claim 2, characterized in that, The scheduling system is specifically configured as follows: When the target task type is a first command script type, the first local operator corresponding to the first command script type is determined as the target operator, and different command script types correspond to different operators; When the target task type is a first ETL type, the second local operator corresponding to the first ETL type is determined as the target operator, and different ETL types correspond to different operators.

4. The task processing device according to claim 1, characterized in that, When the task type included in the configuration information is a command script type, the task execution parameters corresponding to the configuration information include command script parameters; When the task type included in the configuration information is ETL, the task execution parameters corresponding to the configuration information include the data source parameters and data destination parameters of the corresponding task node.

5. The task processing device according to claim 1, characterized in that, The task processing device also includes a lineage analysis system, configured as follows: If the configuration information also includes lineage information of the corresponding task nodes, then based on the multiple configuration information, obtain the full-link lineage information corresponding to the multiple task nodes; If the configuration information does not include the lineage information of the corresponding task node, during the execution of the DAG instance, the full-link lineage information of the multiple task nodes is obtained by parsing the task execution logic corresponding to each task node. The full-link lineage information includes the intra-node lineage relationships of each task node and the data flow lineage relationships between the nodes corresponding to the multiple task nodes.

6. The task processing device according to claim 5, characterized in that, The bloodline analysis system is specifically configured as follows: When the target task node is a command script type task, the target lineage information of the target task node is obtained by parsing the target command script of the target task node. The target lineage information includes the intra-node lineage relationship of the target task node and the data flow lineage relationship between the nodes corresponding to the target task node. When the target task node is an ETL type task, the intra-node lineage of the target task node and the data flow lineage between the nodes corresponding to the target task node are obtained by parsing the target SQL statement of the target task node.

7. The task processing device according to claim 6, characterized in that, The bloodline analysis system is specifically configured as follows: When the target task node is a command script type task, obtain the target execution command corresponding to the target task node; If the target execution command matches the first preset execution command among a plurality of preset execution commands, the lineage information corresponding to the first preset execution command is determined as the target lineage information; If the target execution command does not match any of the multiple preset execution commands, the target execution command is initially parsed to obtain script-related information of the target command script. Based on the script-related information, the target command script is obtained from the file system; The target command script is parsed to obtain the script parsing result; Based on the script parsing results, the target bloodline information is determined.

8. The task processing device according to any one of claims 1-7, characterized in that, The scheduling system is also configured as follows: Before executing the DAG instance, for the target task instance in the DAG instance, obtain multiple load information corresponding to multiple service nodes respectively. The load information includes the computing load utilization, data load throughput and data transmission latency of the corresponding service node. Based on the multiple load information, determine the comprehensive score corresponding to each service node; The target task instance is distributed to the target service node with the highest overall score, so that the corresponding heterogeneous execution engine on the target service node executes the target task instance.

9. The task processing device according to claim 8, characterized in that, The scheduling system is specifically configured as follows: Based on the task type of the target task instance, the corresponding target comprehensive score formula is determined, and different task types correspond to different comprehensive score formulas. Based on the multiple load information and the target comprehensive score formula, the comprehensive score corresponding to each service node is determined.

10. A task processing method, characterized in that, Applied to task processing devices, including: Display the task configuration page; In response to the user's input operation on the task configuration page, multiple configuration information corresponding to multiple task nodes are obtained. The configuration information includes the task type, task dependency relationship and task execution parameters of the corresponding task node. The multiple configuration information is parsed to generate a directed acyclic graph (DAG) topology, which includes the multiple task nodes and their execution order. Based on the DAG topology, generate a DAG instance; Execute the DAG instance.

Images