Data processing task execution method and device, storage medium and equipment

Abstract

The invention provides a data processing task execution method and device, a storage medium and equipment, and the method comprises the steps: receiving a data processing task demand submitted by a user, carrying out the analysis of the task processing demand, obtaining an analysis result, automatically building a DAG model according to the analysis result through employing a large model technology, and carrying out the execution of a data processing task according to the analysis result. And querying the metadata knowledge graph to obtain a metadata driving mechanism related to an analysis result, and automatically scheduling an execution sequence of nodes in the DAG model according to the metadata driving mechanism to obtain an execution result of the data processing task. Through the metadata driving mechanism, automatic scheduling of the node execution sequence in the DAG model is achieved, manual intervention is not needed, the automation degree of task execution is improved, and through the metadata driving mechanism, the accuracy of node information and the dependency relationship is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, storage medium and equipment for executing a data processing task. Background Art

[0002] With the evolution of big data technology, the data middle platform system has been increasingly applied in various business fields. The task scheduling system, as the foundation of the data middle platform system, carries the task scheduling, distribution and execution of the entire data middle platform. Therefore, how to optimize the existing traditional task scheduling system to improve its ease of use for developers and its own task execution efficiency is an urgent task.

[0003] The current traditional task scheduling system generally only supports the execution of the overall task in the order of the directed acyclic graph (DAG). The condition for whether a task node will be triggered to execute is that all its upstream nodes are triggered to execute, and the execution results are all successful. This task execution logic can meet some development scenarios, but it does not take into account the scenario that, for example, there are multiple upstream nodes of a certain task node N that are not executed or failed to execute, but node N still needs to be executed. In other words, developers cannot set the logic of whether the node is triggered to execute or how it is triggered to execute. That is, the execution logic setting of the task node in the relevant technology is not flexible enough.

[0004] Currently, no effective solution has been proposed to the problem of poor flexibility in setting the execution logic of task nodes in the related art. Summary of the invention

[0005] In view of this, the present invention provides a method, apparatus, storage medium and device for executing data processing tasks, which can dynamically build a DAG model based on task requirements, and finely control and optimize the task execution process through metadata, thereby significantly improving the execution efficiency and accuracy of data processing tasks.

[0006] In a first aspect, an embodiment of the present invention provides a method for executing a data processing task, the method comprising:

[0007] Receiving data processing task requirements submitted by users, and parsing the task processing requirements to obtain parsing results;

[0008] According to the analysis results, a DAG model is automatically constructed using a large model technology;

[0009] According to the parsing result, query the metadata knowledge graph to obtain a metadata driving mechanism related to the parsing result;

[0010] According to the metadata driven mechanism, the execution order of the nodes in the DAG model is automatically scheduled to obtain the execution result of the data processing task.

[0011] Furthermore, the receiving of the data processing task requirement proposed by the user and parsing the task processing requirement to obtain the parsing result includes:

[0012] Receive data processing task requirements submitted by users;

[0013] The data processing task requirements are parsed using grammatical parsing and semantic understanding methods to obtain the task objectives of the data processing task and the dependencies between the task objectives.

[0014] Furthermore, according to the analysis result, automatically constructing the DAG model using the large model technology includes:

[0015] Constructing nodes of a DAG model according to the task objectives;

[0016] Adding directed edges to the DAG model according to the dependency relationship between the task objectives;

[0017] Use the topological sorting algorithm to prioritize the task objectives and confirm the execution order of the task objectives.

[0018] Furthermore, while adding directed edges in the DAG model according to the dependency relationship between the task objectives, the method further includes:

[0019] Check whether there is a loop in the DAG model;

[0020] When there is no loop in the DAG model, a topological sorting algorithm is used to prioritize the task objectives and confirm the sorting results of the task objectives.

[0021] Further, according to the parsing result, querying the metadata knowledge graph to obtain a metadata driving mechanism related to the parsing result includes:

[0022] Constructing a query statement for querying the metadata knowledge graph according to the task objectives and the dependencies between the task objectives;

[0023] Submit the constructed query statement to the metadata knowledge graph to execute the query and obtain the query results. Analyze the query results to extract the metadata driving mechanism related to the parsing results.

[0024] Furthermore, according to the metadata driven mechanism, automatically scheduling the execution order of the nodes in the DAG model to obtain the execution result of the data processing task includes:

[0025] According to the sorting results of the task objectives, the execution order of the nodes is triggered in sequence;

[0026] During the execution process, the metadata driven mechanism automatically schedules the execution order of the nodes in the DAG model.

[0027] Furthermore, during the execution process, the metadata-driven mechanism automatically schedules the execution order of the nodes in the DAG model, including:

[0028] When a node fails to execute, exception handling is performed according to the fault tolerance mechanism in the metadata-driven mechanism, and the exception handling at least includes: retrying the execution, skipping the failed node, or triggering an alarm, etc.; or

[0029] According to the metadata-driven mechanism, when a task target trigger condition is met, the task target is automatically triggered for execution; or

[0030] When the metadata changes, dynamically adjust the execution order of the nodes in the DAG model according to the change of the metadata; or

[0031] A conditional judgment node is added in the DAG model, and the next node to be executed is selected according to the value or state of the metadata in the metadata driving mechanism.

[0032] In a second aspect, an embodiment of the present invention provides a device for executing a data processing task, the device comprising:

[0033] The receiving and analyzing module is used to receive the data processing task requirements submitted by the user and analyze the task processing requirements to obtain the analysis results;

[0034] A DAG model construction module, used to automatically construct a DAG model using a large model technology according to the analysis results;

[0035] A metadata driving mechanism acquisition module, used to query the metadata knowledge graph to acquire a metadata driving mechanism related to the analysis result according to the analysis result;

[0036] An execution module is used to automatically schedule the execution order of the nodes in the DAG model according to the metadata driving mechanism to obtain the execution result of the data processing task.

[0037] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to execute any one of the methods in the first aspect when executed.

[0038] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any one of the methods described in the first aspect.

[0039] The technical solution provided by the present invention has the following four beneficial effects. First, the solution makes data processing tasks efficient and automated. Through the metadata-driven mechanism, the automatic scheduling of the execution order of nodes in the DAG model is realized without manual intervention, which greatly improves the automation of task execution. The nodes are sorted using a topological sorting algorithm to ensure that all predecessor nodes of each node are ranked before the node, thereby achieving efficient execution of tasks. At the same time, the solution also supports parallel processing, which can make full use of computing resources and improve the throughput of task processing. Second, flexibility and scalability. The solution can easily add, delete or modify nodes and dependencies to adapt to changing needs. This makes the DAG model very flexible and can handle various complex data processing tasks. With the development of business and the growth of data, the solution can easily expand the scale and complexity of the DAG model to meet higher data processing requirements. At the same time, the solution also supports integration with other systems or tools to achieve a wider range of application scenarios. Thirdly, high accuracy and high reliability. The solution ensures the accuracy of node information and dependencies through the metadata-driven mechanism. During the task execution process, the execution status and resource usage of the nodes can be monitored in real time, and abnormal situations can be discovered and handled in time to ensure the correctness of task execution. The solution has established a complete fault-tolerant mechanism, such as retry execution, skipping failed nodes, triggering alarms and other strategies to deal with possible abnormal situations. This ensures that even if some nodes fail during the task execution, it will not affect the stability and reliability of the entire system. Fourthly, good maintainability and strong optimization. The solution realizes unified management of DAG models and nodes through the metadata-driven mechanism. This reduces the difficulty of system maintenance and upgrades, making the system easier to maintain and manage. The solution can optimize the DAG model and metadata based on performance analysis and optimization strategies during execution. Through iterative updates, the efficiency and accuracy of task execution can be continuously improved, achieving continuous optimization and improvement of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a method for executing a data processing task provided in Embodiment 1 of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of a device for executing a data processing task provided in Embodiment 2 of the present invention;

[0042] Figure 3It is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1

[0045] See also Figure 1 , Figure 1 1 is a flowchart of a method for executing a data processing task provided by an embodiment of the present invention, the method comprising the following steps:

[0046] Step 11: Receive the data processing task requirements submitted by the user, and parse the task processing requirements to obtain parsing results.

[0047] In this step, the system first receives the data processing task requirements submitted by the user, which include but are not limited to task types (such as data cleaning, data conversion, data analysis, etc.), input data (data source, data format, etc.), and output requirements (output format, output location, etc.). By parsing these requirements, the system clarifies the goals and constraints of the task. The parsing process includes steps such as grammatical analysis and semantic understanding to ensure accurate understanding of user intent.

[0048] In some embodiments, step 11 may be implemented by:

[0049] Step 111: Receive data processing task requirements submitted by users.

[0050] In this step, the system receives data processing task requirements submitted by users. Users may submit their task requirements through a graphical user interface (GUI), command line interface (CLI), or other methods (such as API calls). These requirements may include the data set to be processed, the expected output results, and specific requirements for data conversion or analysis.

[0051] The user may provide input in the form of natural language, structured query language (such as SQL), or application-specific scripts or codes. The system may verify the legality and completeness of the input to ensure that the user has provided sufficient information to perform the data processing task.

[0052] Step 112: parse the data processing task requirements using syntax parsing and semantic understanding methods to obtain task objectives of the data processing task and dependencies between task objectives.

[0053] In this step, the system uses grammatical parsing and semantic understanding methods to process the task requirements submitted by the user to obtain the task objectives and the dependencies between task objectives.

[0054] The system first parses the grammatical structure of the user input. This usually involves converting the input into an internal representation (such as an abstract syntax tree) that the system can understand and process. Syntax parsing ensures that the user input conforms to the expected format and rules.

[0055] Based on grammatical parsing, the system further understands the semantic content of the user input. This includes identifying task objectives (such as data cleaning, transformation, aggregation, etc.), dependencies between task objectives (such as which tasks must be performed after other tasks), and any related parameters or settings.

[0056] Through semantic understanding, the system can build a dependency graph (such as DAG) between task goals. This graph shows which tasks need to be executed first and which tasks can be processed in parallel.

[0057] These two steps together constitute the complete process of receiving and parsing data processing task requirements. By receiving the task requirements input by the user and parsing them using grammatical parsing and semantic understanding methods, the system can accurately identify the task objectives and the dependencies between them, thus providing a basis for subsequent task execution and scheduling.

[0058] It should be noted that, in other embodiments, the reception and analysis of data processing task requirements may also require other steps, such as data preprocessing, task execution, result generation and output, etc.

[0059] Step 12: According to the analysis results, a DAG model is automatically constructed using large model technology.

[0060] The DAG model is a model that uses a directed acyclic graph to represent the dependencies between tasks. In a directed acyclic graph, each node represents a task or operation, and the directed edges between nodes represent the dependencies between tasks. This representation method enables the DAG model to clearly display the execution order and dependency hierarchy between tasks, providing strong support for task scheduling and execution.

[0061] In some embodiments, step 12 may be implemented by:

[0062] Step 121: construct nodes of a DAG model according to the task objectives.

[0063] In this step, the system constructs nodes of the DAG model based on the parsed task objectives. Each task objective is considered a node in the DAG, which represents a specific operation or calculation that needs to be performed. For each task objective, the system creates a corresponding DAG node. These nodes usually contain necessary information about the task objectives, such as task type, input data, output data, etc. Each node may also have some attributes, such as execution time, resource requirements (such as memory and CPU), priority, etc. These attributes can help the system make decisions in the subsequent task scheduling and execution process.

[0064] Step 122: Add directed edges to the DAG model according to the dependency relationship between the task objectives.

[0065] In this step, the system constructs directed edges in the DAG based on the dependencies between task targets. These edges represent the order between tasks, that is, which tasks must be executed after other tasks.

[0066] First, the system determines which tasks have a sequence by analyzing the dependencies between task targets. These dependencies may come from user input, logical requirements for data processing, etc. Then, for each pair of task targets with dependencies, the system adds a directed edge from the first task to the second task in the DAG. These edges ensure the correct execution order between tasks.

[0067] Step 123: Use a topological sorting algorithm to prioritize the task objectives and confirm the execution order of the task objectives.

[0068] In this step, the system uses a topological sorting algorithm to sort the nodes in the DAG to determine the execution order of the task objectives. Topological sorting is a sorting algorithm for DAG that generates a linear sequence in which each node is arranged according to its topological order in the DAG. In this sequence, for any directed edge (u, v), node u is ranked before node v. Through topological sorting, the system can obtain one or more valid task execution orders. These orders ensure that all task objectives can be executed correctly according to their dependencies. If there are multiple valid execution orders, the system may choose the best order based on other factors (such as resource utilization, execution time, etc.).

[0069] In other embodiments of the present invention, the constructed DAG model can also be optimized, for example: Parallel processing: In the DAG model, find tasks that can be executed in parallel to improve the overall execution efficiency. For example, in the machine learning training process, data preprocessing tasks and model training tasks can be executed in parallel. Resource allocation: According to the requirements of the task and the resource conditions of the system, reasonably allocate resources to each task. This can be achieved through the task scheduling system to ensure that each task can obtain sufficient resources for efficient execution.

[0070] These three steps together constitute the complete process from task objectives to DAG model construction to task execution order determination. Through this process, the system can accurately identify the dependencies between task objectives and generate an effective task execution order, thus providing a basis for subsequent task scheduling and execution.

[0071] In some other embodiments of the present invention, after step 12 and before step 13, the method may further include:

[0072] Step 12a: Check whether there is a loop in the DAG model.

[0073] An important feature of the DAG model is that it does not contain loops. The existence of loops will cause the dependencies between tasks to form a closed loop, making it impossible for tasks to be executed in a certain order. Therefore, before sorting the task goals in the DAG model, you must first check whether there are loops in the model.

[0074] Loop detection algorithm: Usually, algorithms such as depth-first search (DFS) or Kahn's algorithm (also known as the preprocessing step of topological sorting) can be used to detect whether there are loops in the DAG model. These algorithms traverse the nodes in the graph and check whether there is a path from a node to the node again.

[0075] Result processing: If a loop is detected, the system needs to take appropriate measures, such as prompting the user to check and correct the DAG model, or automatically trying to adjust the model structure to eliminate the loop.

[0076] Step 12b: When there is no loop in the DAG model, use a topological sorting algorithm to prioritize the task objectives and confirm the sorting results of the task objectives.

[0077] After ensuring that the DAG model does not contain loops, the next step is to prioritize the task objectives using a topological sorting algorithm. Topological sorting is a linear sorting that determines the execution order of task objectives based on the dependencies in the DAG model.

[0078] Topological sorting algorithm: Commonly used topological sorting algorithms include DFS-based recursive algorithms and in-degree-based iterative algorithms. These algorithms traverse the nodes in the DAG model and determine their execution order based on the node dependencies (i.e. predecessor and successor nodes).

[0079] Sorting result: The result of topological sorting is a linear sequence, which represents the execution order of task objectives determined by dependencies. This sequence can serve as the basis for subsequent task scheduling and execution.

[0080] Step 12a and step 12b together constitute the key process in DAG model processing. First, the validity of the DAG model is ensured by checking the loop; then, the task objectives are prioritized using a topological sorting algorithm to determine their execution order. These two steps are crucial to ensure that the tasks in the DAG model can be executed in the correct order.

[0081] Step 13: According to the analysis result, query the metadata knowledge graph to obtain the metadata driving mechanism related to the analysis result.

[0082] Metadata is data that describes data. It provides information about the data, such as the source, structure, and meaning of the data. Metadata plays a vital role in data management. It can help users understand, discover, manage, and use data. Metadata-driven uses metadata to describe and drive the use and flow of data. Through metadata, the system can automatically process data, generate reports, trigger processes, etc.

[0083] In some embodiments, step 13 may be implemented by:

[0084] Step 131: construct a query statement for querying the metadata knowledge graph based on the task objectives and the dependencies between the task objectives.

[0085] In this step, the system constructs a query statement for the metadata knowledge graph based on the task goals and the dependencies between them. This query statement is designed to extract information related to the task goals and dependencies from the knowledge graph. The system first determines the type of information to be queried, which may include the attributes of the dataset, the location of the data source, the rules for data transformation, etc. Based on these query goals, the system uses an appropriate query language (such as SPARQL for RDF knowledge graphs) to construct a query statement. This statement explicitly specifies the entities, attributes, and relationships between them to match the task goals and dependencies.

[0086] Step 132: Submit the constructed query statement to the metadata knowledge graph to execute the query and obtain the query results.

[0087] In this step, the system submits the constructed query statement to the metadata knowledge graph and waits for the execution result. The system sends the query statement to the metadata knowledge graph through an appropriate interface (such as RESTAPI, GraphQL, etc.). After the knowledge graph processes the query statement, it returns a result set. This result set may contain multiple matches, each of which provides information related to the query target.

[0088] Step 133: Analyze the query results to extract metadata driving mechanisms related to the parsing results.

[0089] In this step, the system conducts in-depth analysis of the query results to extract metadata-driven mechanisms that are directly related to the task goals and dependencies.

[0090] The system first parses the query results and converts them into an internal representation for further analysis. By analyzing the information in the result set, the system can identify metadata-driven mechanisms related to task goals and dependencies. These mechanisms may include data flow paths, data transformation rules, data quality checkpoints, etc. Finally, the system integrates these extracted metadata-driven mechanisms into one or more models for use in subsequent data processing tasks.

[0091] These three steps together constitute a complete process from task objectives and dependencies to metadata knowledge graph query and then to metadata driving mechanism extraction. Through this process, the system can automatically extract information related to data processing tasks from the knowledge graph and use this information to guide subsequent data processing operations.

[0092] Step 14: According to the metadata driven mechanism, the execution order of the nodes in the DAG model is automatically scheduled to obtain the execution result of the data processing task.

[0093] Based on the constructed DAG model and metadata-driven mechanism, the system automatically schedules execution nodes. The scheduling process includes steps such as resource allocation and task priority sorting.

[0094] During the execution process, the system monitors the execution status of the task, including the execution progress of the node, resource usage, etc. The monitoring process is carried out in real time to ensure that problems are discovered and handled in a timely manner. Once a problem or abnormal situation is found (such as node execution failure, insufficient resources, etc.), the system will automatically adjust and optimize. Adjustment measures include reallocating resources, adjusting the execution order, triggering the exception handling process, etc. Through continuous monitoring and adjustment, the system ensures that the task can be completed smoothly and outputs results that meet user requirements.

[0095] In some embodiments, step 14 may be implemented by:

[0096] Step 141: trigger the execution order of the nodes in sequence according to the sorting results of the task objectives.

[0097] In this step, the system will trigger the execution of nodes in the DAG model in sequence according to the task target execution order obtained previously through topological sorting.

[0098] The system first obtains the sorting results of the task targets. This sorting result ensures that all tasks can be executed correctly according to their dependencies. According to the sorting results, the system triggers the nodes in the DAG in sequence. This means that for each node, the system checks whether it meets the execution conditions (for example, whether all the predecessor nodes have been executed). Once the conditions are met, the system triggers the execution of the node.

[0099] Step 142: During the execution process, the metadata-driven mechanism automatically schedules the execution order of the nodes in the DAG model.

[0100] In this step, the metadata-driven mechanism plays a key role in the node execution process. It is responsible for automatically scheduling the execution order of nodes in the DAG model.

[0101] The metadata-driven mechanism uses information extracted from the metadata knowledge graph to guide the execution of nodes. This information may include the attributes of the data set, the location of the data source, the rules for data conversion, etc.

[0102] During execution, the metadata-driven mechanism dynamically schedules the execution of subsequent nodes based on the execution status and dependencies of the current node. If a node fails to execute or requires additional processing time, the metadata-driven mechanism can adjust the execution plan of subsequent nodes accordingly. In addition, the metadata-driven mechanism can also consider the system's resource usage (such as CPU, memory, disk space, etc.) to optimize the execution order of nodes. For example, it may prioritize the execution of nodes that have lower resource requirements or can be processed in parallel.

[0103] These two steps together constitute the complete process from the task target sorting results to the DAG model node execution order triggering, and emphasize the important role of the metadata-driven mechanism in the execution process. Through this process, the system can ensure that all tasks can be correctly executed according to their dependencies, and make full use of the information in the metadata knowledge graph to optimize the node execution order and resource usage.

[0104] In some embodiments, the metadata-driven mechanism automatically schedules the execution order of nodes in the DAG model, which may include the following situations:

[0105] Scenario 1: When a node fails to execute, exception handling is performed according to the fault tolerance mechanism in the metadata-driven mechanism. The exception handling at least includes: retrying the execution, skipping the failed node, or triggering an alarm.

[0106] When a node in the DAG model fails to execute, the fault tolerance mechanism in the metadata-driven mechanism will intervene to handle the exception. This exception handling includes at least the following strategies:

[0107] Retry execution: The system automatically retries failed execution nodes according to the configured retry strategy (such as the number of retries, retry interval, etc.). This helps to handle execution failures caused by temporary network problems, resource competition, or system load.

[0108] Skip the failed node: In some cases, if the failure of a node does not affect the final result of the entire DAG model or the execution of subsequent nodes, the system can choose to skip the node. This can prevent the entire DAG model from stagnating due to the failure of a single node.

[0109] Trigger an alarm: The system will notify relevant personnel or system administrators through the configured alarm mechanism (such as sending emails, text messages, triggering alarms in the monitoring system, etc.) so that they can intervene in time to handle the failed nodes.

[0110] Scenario 2: According to the metadata-driven mechanism, when a task target trigger condition is met, the task target is automatically triggered for execution.

[0111] The metadata-driven mechanism can monitor the trigger conditions of each task target in the DAG model. When the trigger conditions of a task target are met, the system will automatically trigger the execution of the task target. This trigger mechanism can be based on multiple conditions such as time, events, data status, etc., to ensure that the tasks in the DAG model can be correctly executed according to the predetermined logic and timing.

[0112] Scenario 3: When the metadata changes, the execution order of the nodes in the DAG model is dynamically adjusted according to the change of the metadata.

[0113] When metadata changes, the metadata-driven mechanism can perceive these changes in real time and dynamically adjust the execution order of nodes in the DAG model based on these changes. This dynamic adjustment ensures that the DAG model can adapt to changes in the data environment or business logic and maintain efficient and accurate execution.

[0114] To achieve this dynamic adjustment, the system needs to: Monitor metadata changes in real time: Use the metadata monitoring mechanism to capture metadata changes in real time, including data addition, deletion, and modification. Analyze the impact of metadata changes on the DAG model: Based on metadata changes, analyze the potential impact of these changes on the execution order of nodes in the DAG model. Dynamically adjust the execution order: Based on the analysis results, dynamically adjust the execution order of nodes in the DAG model to ensure that tasks can be correctly executed according to the new logic and timing.

[0115] Scenario 4: Add a conditional judgment node in the DAG model, and select the next node to be executed according to the value or state of the metadata in the metadata driving mechanism.

[0116] Adding conditional nodes to the DAG model is an effective way to flexibly handle the order of task execution. These conditional nodes select the next node to be executed based on the value or state of the metadata in the metadata-driven mechanism.

[0117] Configure conditional judgment nodes: In the DAG model, configure appropriate conditional judgment nodes according to the needs of business logic and data flow.

[0118] Set judgment conditions and branches: Set judgment conditions and corresponding execution branches for each conditional judgment node. These conditions can be based on the value, state, or relationship with other nodes of the metadata.

[0119] Dynamically select execution paths: At runtime, the conditional judgment node will determine which execution path should be selected based on the value or status of the current metadata. This dynamic selection ensures that the DAG model can adapt to different data environments and business scenarios.

[0120] From the above description, we can see that the metadata-driven mechanism plays a vital role in the execution of the DAG model. It not only ensures that tasks are correctly executed according to the predetermined logic and timing, but also dynamically adjusts the execution order according to the changes in metadata and handles various abnormal situations, thereby maintaining the efficiency and stability of the entire DAG model.

[0121] In other embodiments of the present invention, the execution status of the DAG model may also be monitored through visualization tools (such as Apache Superset, Grafana, etc.). These tools generally provide functions such as real-time update, task progress bar, log output and error reporting, so that system administrators can find and solve problems in a timely manner.

[0122] The technical solution provided by the embodiment of the present invention has the following four beneficial effects. First, the solution makes data processing tasks efficient and automated. Through the metadata-driven mechanism, the automatic scheduling of the execution order of nodes in the DAG model is realized without manual intervention, which greatly improves the automation of task execution. The nodes are sorted by the topological sorting algorithm to ensure that all the predecessor nodes of each node are arranged before the node, thereby realizing efficient execution of tasks. At the same time, the solution also supports parallel processing, which can make full use of computing resources and improve the throughput of task processing. Second, flexibility and scalability. The solution can easily add, delete or modify nodes and dependencies to adapt to changing needs. This makes the DAG model very flexible and can cope with various complex data processing tasks. With the development of business and the growth of data, the solution can easily expand the scale and complexity of the DAG model to meet higher data processing requirements. At the same time, the solution also supports integration with other systems or tools to achieve a wider range of application scenarios. Third, high accuracy and high reliability. The solution ensures the accuracy of node information and dependencies through the metadata-driven mechanism. During the task execution process, the execution status and resource usage of the nodes can be monitored in real time, and abnormal situations can be discovered and handled in a timely manner to ensure the correctness of task execution. The solution has established a complete fault-tolerant mechanism, such as retrying execution, skipping failed nodes, triggering alarms and other strategies to deal with possible abnormal situations. This ensures that even if some nodes fail during the task execution, it will not affect the stability and reliability of the entire system; fourthly, the maintainability and optimization are good. The solution realizes the unified management of DAG models and nodes through the metadata-driven mechanism. This reduces the difficulty of system maintenance and upgrades, making the system easier to maintain and manage. The solution can optimize the DAG model and metadata based on the performance analysis and optimization strategies during the execution process. Through iterative updates, the efficiency and accuracy of task execution can be continuously improved, and the system can be continuously optimized and improved.

[0123] Embodiment 2

[0124] See also Figure 2 , Figure 2 : is a schematic diagram of a structure diagram of a data processing task device provided in Embodiment 2 of the present invention, wherein the device comprises:

[0125] The receiving and analyzing module 21 is used to receive the data processing task requirements submitted by the user, and analyze the task processing requirements to obtain the analysis results;

[0126] A DAG model building module 22, used to automatically build a DAG model using a large model technology according to the analysis results;

[0127] A metadata driving mechanism acquisition module 23 is used to query the metadata knowledge graph to acquire a metadata driving mechanism related to the analysis result according to the analysis result;

[0128] The execution module 24 is used to automatically schedule the execution order of the nodes in the DAG model according to the metadata driving mechanism to obtain the execution result of the data processing task.

[0129] In some embodiments of the present invention, the receiving and parsing module 21 may include:

[0130] A receiving unit 211 is used to receive a data processing task requirement submitted by a user;

[0131] The parsing unit 212 parses the data processing task requirements using grammatical parsing and semantic understanding methods to obtain task objectives of the data processing task and dependencies between task objectives.

[0132] In some embodiments of the present invention, the DAG model building module 22 may include:

[0133] A node construction unit 221 is used to construct a node of a DAG model according to the task target;

[0134] A directed edge adding unit 222, configured to add directed edges in the DAG model according to the dependency relationship between the task objectives;

[0135] The sorting unit 223 is used to use a topological sorting algorithm to prioritize the task objectives and confirm the execution order of the task objectives.

[0136] In some other embodiments of the present invention, the device may further include:

[0137] The judgment module 22a is used to check whether there is a loop in the DAG model. When there is no loop in the DAG model, a topological sorting algorithm is used to prioritize the task objectives and confirm the sorting result of the task objectives.

[0138] In some other embodiments of the present invention, the metadata drive mechanism acquisition module 23 may include:

[0139] A query statement construction unit 231 is used to construct a query statement for querying the metadata knowledge graph according to the task objectives and the dependencies between the task objectives;

[0140] The query result acquisition unit 232 is used to submit the constructed query statement to the metadata knowledge graph to execute the query and obtain the query result;

[0141] The analysis and extraction unit 233 is used to analyze the query result to extract the metadata driving mechanism related to the parsing result.

[0142] In some other embodiments of the present invention, the execution module 24 may include:

[0143] The trigger unit 241 is used to trigger the execution order of the nodes in sequence according to the sorting result of the task objectives;

[0144] The automatic scheduling unit 242 is used to automatically schedule the execution order of the nodes in the DAG model by the metadata-driven mechanism during the execution process.

[0145] The automatic scheduling unit 242 may be configured to perform the following operations:

[0146] When a node fails to execute, exception handling is performed according to the fault tolerance mechanism in the metadata-driven mechanism, and the exception handling at least includes: retrying the execution, skipping the failed node, or triggering an alarm, etc.; or

[0147] According to the metadata-driven mechanism, when a task target trigger condition is met, the task target is automatically triggered for execution; or

[0148] When the metadata changes, dynamically adjust the execution order of the nodes in the DAG model according to the change of the metadata; or

[0149] A conditional judgment node is added in the DAG model, and the next node to be executed is selected according to the value or state of the metadata in the metadata driving mechanism.

[0150] Therefore, the technical solution provided by the embodiment of the present invention has the following four beneficial effects. First, the solution makes the data processing task efficient and automatic. Through the metadata-driven mechanism, the automatic scheduling of the execution order of nodes in the DAG model is realized without manual intervention, which greatly improves the automation of task execution. The nodes are sorted by the topological sorting algorithm to ensure that all the predecessor nodes of each node are arranged before the node, thereby realizing efficient execution of the task. At the same time, the solution also supports parallel processing, which can make full use of computing resources and improve the throughput of task processing. Second, flexibility and scalability. The solution can easily add, delete or modify nodes and dependencies to adapt to changing needs. This makes the DAG model very flexible and can cope with various complex data processing tasks. With the development of business and the growth of data, the solution can easily expand the scale and complexity of the DAG model to meet higher data processing requirements. At the same time, the solution also supports integration with other systems or tools to achieve a wider range of application scenarios. Third, high accuracy and high reliability. The solution ensures the accuracy of node information and dependencies through the metadata-driven mechanism. During the task execution process, the execution status and resource usage of the nodes can be monitored in real time, and abnormal situations can be discovered and handled in a timely manner to ensure the correctness of task execution. The solution has established a complete fault-tolerant mechanism, such as retrying execution, skipping failed nodes, triggering alarms and other strategies to deal with possible abnormal situations. This ensures that even if some nodes fail during the task execution, it will not affect the stability and reliability of the entire system; fourthly, the maintainability and optimization are good. The solution realizes the unified management of DAG models and nodes through the metadata-driven mechanism. This reduces the difficulty of system maintenance and upgrades, making the system easier to maintain and manage. The solution can optimize the DAG model and metadata based on the performance analysis and optimization strategies during the execution process. Through iterative updates, the efficiency and accuracy of task execution can be continuously improved, and the system can be continuously optimized and improved.

[0151] It should be noted that the device for data processing tasks in the embodiment of the present invention and the method for data processing tasks in the above-mentioned embodiment belong to the same inventive concept. The technical details not described in detail in this device can be found in the previous description of the method and will not be repeated here.

[0152] In addition, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the aforementioned method when running.

[0153] Figure 3A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0154] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0155] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as an idle detection method.

[0157] In some embodiments, the idle detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the idle detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the idle detection method in any other appropriate manner (e.g., by means of firmware).

[0158] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0160] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0162] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0163] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0164] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0165] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for executing a data processing task, characterized in that: The method comprises: Receiving data processing task requirements submitted by users, and parsing the task processing requirements to obtain parsing results; According to the analysis results, a DAG model is automatically constructed using a large model technology; According to the parsing result, query the metadata knowledge graph to obtain a metadata driving mechanism related to the parsing result; According to the metadata driven mechanism, the execution order of the nodes in the DAG model is automatically scheduled to obtain the execution result of the data processing task.

2. The method according to claim 1, characterized in that The receiving of the data processing task requirement proposed by the user and parsing the task processing requirement to obtain the parsing result includes: Receive data processing task requirements submitted by users; The data processing task requirements are parsed using grammatical parsing and semantic understanding methods to obtain the task objectives of the data processing task and the dependencies between the task objectives.

3. The method according to claim 2, characterized in that According to the analysis results, the DAG model is automatically constructed using the large model technology, including: Constructing nodes of a DAG model according to the task objectives; Adding directed edges to the DAG model according to the dependency relationship between the task objectives; Use the topological sorting algorithm to prioritize the task objectives and confirm the execution order of the task objectives.

4. The method according to claim 3, characterized in that While adding directed edges in the DAG model according to the dependency relationship between the task objectives, the method further includes: Check whether there is a loop in the DAG model; When there is no loop in the DAG model, a topological sorting algorithm is used to prioritize the task objectives and confirm the sorting results of the task objectives.

5. The method according to claim 2, characterized in that: According to the parsing result, querying the metadata knowledge graph to obtain metadata driving mechanism related to the parsing result includes: Constructing a query statement for querying the metadata knowledge graph according to the task objectives and the dependencies between the task objectives; Submit the constructed query statement to the metadata knowledge graph to execute the query and obtain the query results; The query results are analyzed to extract metadata related to the parsing results.

6. The method according to claim 4, characterized in that Automatically scheduling the execution order of the nodes in the DAG model according to the metadata-driven mechanism to obtain the execution result of the data processing task includes: According to the sorting results of the task objectives, the execution order of the nodes is triggered in sequence; During the execution process, the metadata driven mechanism automatically schedules the execution order of the nodes in the DAG model.

7. The method according to claim 6, characterized in that During the execution process, the metadata-driven mechanism automatically schedules the execution order of the nodes in the DAG model, including: When a node fails to execute, exception handling is performed according to the fault tolerance mechanism in the metadata-driven mechanism, and the exception handling at least includes: retrying the execution, skipping the failed node, or triggering an alarm, etc.; or According to the metadata-driven mechanism, when a task target trigger condition is met, the task target is automatically triggered for execution; or When the metadata changes, dynamically adjust the execution order of the nodes in the DAG model according to the change of the metadata; or A conditional judgment node is added in the DAG model, and the next node to be executed is selected according to the value or state of the metadata in the metadata driving mechanism.

8. A data processing task execution device, characterized in that: The device comprises: The receiving and analyzing module is used to receive the data processing task requirements submitted by the user and analyze the task processing requirements to obtain the analysis results; A DAG model construction module, used to automatically construct a DAG model using a large model technology according to the analysis results; A metadata driving mechanism acquisition module, used to query the metadata knowledge graph to acquire a metadata driving mechanism related to the analysis result according to the analysis result; The execution module is used to automatically schedule the execution order of the nodes in the DAG model according to the metadata driving mechanism to obtain the execution result of the data processing task.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Images