Task processing method and device, equipment, storage medium and program product

Abstract

The invention provides a task processing method and device, equipment, a storage medium and a program product. Relates to the technical field of task processing. The method comprises the following steps: analyzing a task set list to determine each task and a to-be-executed command line; detecting the performability of a to-be-executed command line, and performing personalized scheduling on tasks in the task set list according to a service scene of an actual data day settlement task; for the serially executed tasks, respectively executing the command lines of the calling execution programs corresponding to the tasks according to a serial execution sequence, and carrying out serial execution on the tasks; and for the tasks executed in parallel, respectively executing the command lines of the calling execution programs corresponding to the tasks, and executing the tasks in parallel. According to the scheme, personalized scheduling can be performed on the corresponding tasks in the directed acyclic graph according to the specific business scene of the data day settlement task, flexible adjustment of the day settlement sequence is realized, combination of serial processing and parallel processing is realized, and the processing efficiency of the data day settlement task is improved.

Description

Technical Field

[0001] The present application relates to the field of task processing technology, and in particular to a task processing method, device, equipment, storage medium and program product. Background Art

[0002] The data daily settlement task is to process the business data generated in a day, such as processing the data entering the system, the data transferred from the system, and the summary. For example, in the daily settlement task of accounting processing, it is necessary to process the accounting data sent by the accounting engine according to the situation of the upstream accounting entries, including the transfer, summary, transfer of accounts, and value-added tax calculation and transfer.

[0003] At present, the processing of data daily settlement tasks is generally serialized, that is, the processing order of daily settlement is fixed in the system through code, and the data daily settlement tasks are processed serially according to the fixed daily settlement order. However, the serialization of daily settlement steps leads to low efficiency in processing data daily settlement tasks, and technical personnel cannot identify the rationality of serialization and cannot optimize the data daily settlement function. In addition, the fixed daily settlement processing order is difficult to adjust according to the actual business scenario, resulting in low applicability of the processing method. Summary of the invention

[0004] The present application provides a task processing method, apparatus, device, storage medium and program product to flexibly adjust the order of daily settlement tasks according to specific business scenarios.

[0005] In a first aspect, the present application provides a task processing method, comprising: obtaining a task set list from a component database, parsing the task set list, and determining each task and a command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and execution order in all steps of the data daily settlement task, and is generated according to the directed acyclic graph; detecting the executable nature of the command line to be executed, and performing personalized scheduling of the tasks in the task set list according to the actual business scenario of the data daily settlement task, and determining tasks to be executed serially and tasks to be executed in parallel; when the command line to be executed is executable, for the tasks to be executed serially, the command lines for calling the execution program corresponding to each task are executed separately in the order of serial execution, and the tasks are executed serially; for the tasks to be executed in parallel, the command lines for calling the execution program corresponding to each task are executed separately, and the tasks are executed in parallel.

[0006] In the second aspect, the present application provides a task processing device, including: a parsing module, used to obtain a task set list from a component database, parse the task set list, and determine each task and the command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and execution order in all steps of the data daily settlement task, and is generated according to the directed acyclic graph; a verification module, used to detect the executableness of the command line to be executed, and according to the actual business scenario of the data daily settlement task, perform personalized scheduling of the tasks in the task set list, and determine the tasks to be executed serially and the tasks to be executed in parallel; a serial execution module, used to, when the command line to be executed is executable, for the tasks to be executed serially, execute the command line of the calling execution program corresponding to each task in the order of serial execution, and execute the tasks serially; a parallel execution module, used to, for the tasks to be executed in parallel, execute the command line of the calling execution program corresponding to each task, and execute the tasks in parallel.

[0007] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement a task processing method as described in any embodiment of the present application.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the task processing method described in any embodiment of the present application when executed.

[0009] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the task processing method described in any embodiment of the present application.

[0010] The task processing method, device, equipment, storage medium and program product provided by the present application obtain a task set list from a component database, parse the task set list, and determine each task and the command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and execution order in all steps of the data daily settlement task, and is generated according to the directed acyclic graph; the executability of the command line to be executed is detected, and according to the actual business scenario of the data daily settlement task, the tasks in the task set list are personalized and scheduled to determine the tasks to be executed serially and the tasks to be executed in parallel; when the command line to be executed is executable, for the tasks to be executed serially, the command lines for calling the execution program corresponding to each task are respectively executed in the order of serial execution, and the tasks are executed serially; for the tasks to be executed in parallel, the command lines for calling the execution program corresponding to each task are respectively executed, and the tasks are executed in parallel. The technical solution of the embodiment of the present application can perform personalized scheduling of the corresponding tasks in the directed acyclic graph according to the specific business scenario of the data daily settlement task, thereby realizing flexible adjustment of the daily settlement order, and realizing the combination of serial processing and parallel processing, thereby improving the processing efficiency of the data daily settlement task. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1 The process of the task processing method provided in the embodiment of the present application Figure 1 ;

[0013] Figure 2 The process of the task processing method provided in the embodiment of the present application Figure 2 ;

[0014] Figure 3 A flowchart of a specific implementation method of the task processing method provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of the structure of a task processing device provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0017] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0018] The exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. The acquisition, transmission, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0019] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0020] The embodiment of the present application also provides a task processing method. Figure 1 The process of the task processing method provided in the embodiment of the present application Figure 1 The embodiment of the present application can be applied to the case where data daily settlement tasks are processed. The method can be executed by a task processing device, which can be implemented in the form of hardware and / or software, and the task processing device can be configured in an electronic device. Figure 1 As shown, the task processing method includes:

[0021] S110. Obtain a task set list from a component database, parse the task set list, and determine each task and the command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and execution order in all steps of the data daily settlement task, and is generated according to the directed acyclic graph.

[0022] Among them, the inter-group database is a database for storing components needed in the data daily settlement task processing process. The task set list is pre-stored in the component database. The task set list is a directed acyclic graph constructed in advance according to the tasks and execution order in all the steps of the data daily settlement task, and is generated according to the directed acyclic graph. A directed acyclic graph refers to a directed graph without loops. If a directed graph cannot return to the vertex from a vertex through several edges, then the graph is a directed acyclic graph. The tasks in all the steps of the data daily settlement task can be used as nodes, and the execution order between the tasks can be used as the pointing relationship between the nodes to construct a directed acyclic graph. Constructing a directed acyclic graph can make the steps of the data daily settlement task intuitive, and can change the pointing relationship between the nodes to adaptively adjust the execution order of the tasks. The task combination list contains the tasks corresponding to all the steps corresponding to the data daily settlement task, as well as the pending command lines that need to be executed when executing each task. The pending command is the command line that calls the corresponding execution program to implement task processing.

[0023] For example, when there is a data daily settlement task that needs to be processed, a task collection list can be obtained from the component database, and the task collection list can be parsed to determine the tasks that need to be processed for each step of the data daily settlement task contained therein, as well as the command lines to be executed when executing the task.

[0024] S120, detecting the executability of the command line to be executed, and performing personalized scheduling on the tasks in the task set list according to the actual business scenario of the data daily settlement task, and determining the tasks to be executed serially and the tasks to be executed in parallel.

[0025] For example, the executability of the command line to be executed may include the legality of the command line name to be executed, whether there are grammatical problems, whether there are security risks, and abnormal query injection, etc. That is, the executability of the command line to be executed is checked from the form of the command line to be executed.

[0026] Among them, the actual business scenario of the data daily settlement task can be a scenario where systems and customers in different regions perform data daily settlement tasks according to different needs. The specific execution order of the data daily settlement tasks in different business scenarios may be different, so the tasks in the task set list can be personalized according to the specific business scenario of the data daily settlement task to meet the needs of processing the data daily settlement task in the current business scenario. Specifically, the personalized scheduling of the tasks in the task set list can be to determine the tasks that need to be executed serially and the tasks that need to be executed in parallel in the corresponding data daily settlement tasks in the actual business scenario. According to the specific steps of the data daily settlement task in the actual business scenario, the business logic and data dependency between the tasks are determined, and the serial processing order is configured to ensure that the tasks are executed in a normal order and that the subsequent tasks are executed after the predecessor tasks are executed. Tasks that are unrelated to each other and can be processed in parallel can be determined to improve the efficiency and overall performance of task execution, and set appropriate concurrency and resource restrictions. Through the personalized scheduling scheme, it is possible to make adaptive adjustments based on the established directed acyclic graph according to the data daily settlement tasks in the specific business scenario, which is reusable and does not require the construction of program solidification steps for each data daily settlement task.

[0027] S130 . When the command line to be executed is executable, for the tasks to be executed serially, the command lines for calling the execution program corresponding to each task are respectively executed in the order of serial execution, so as to execute the tasks serially.

[0028] Exemplarily, it is a prerequisite to perform an executable detection on the command line to be executed. Only when the command line to be executed is detected to be executable, the command line to be executed will be executed to call the executable program to process the task. In the case that the command line to be executed is executable, for the tasks to be executed serially, the command lines corresponding to the calling execution programs of each task are respectively executed in the order of serial execution to execute the tasks serially. Among them, the execution program is a program for executing the corresponding task, such as functions, commands, etc., and the execution of the execution program can realize the processing of the task. The command line is used to call the execution program. The execution of the command line can call the corresponding execution program to process the task. The order of serial execution can be marked, for example, a serial number is marked for each command line to be executed, and it is executed as needed during execution. The command lines to be executed can also be arranged in the order of the corresponding tasks, and when the command line to be executed is executed, it can be executed in the order of the command line to be executed in this application. Through serial commands, it can be ensured that the tasks are executed in a normal order, and the subsequent tasks are executed after the predecessor tasks are executed.

[0029] S140 , for tasks to be executed in parallel, respectively execute the command lines of the calling execution program corresponding to each task, and execute the tasks in parallel.

[0030] For example, there is no need to limit the execution order between some tasks. It is possible to determine tasks that are not related to each other and can be processed in parallel to improve task execution efficiency and overall performance, and set appropriate concurrency and resource restrictions. According to the step sequence of the data daily settlement task in a specific business scenario, the tasks to be executed in parallel can be determined, and the commands for calling the execution program corresponding to the tasks can be executed separately to execute the tasks in parallel.

[0031] In an embodiment of the present application, the method further includes:

[0032] If there is a task execution failure, repeat the task;

[0033] If the number of times the task is repeatedly executed reaches a first preset number, the name and location of the task that failed to execute are recorded.

[0034] For example, if there is a task execution failure during the execution of a task, in order to verify whether the task execution failure is an accidental event, the task can be repeated to determine whether it can be successfully executed when the task is repeated. If it can be executed successfully, it means that the last task execution failure was an accidental event and can be ignored. If the task still fails when it is repeated, the task is repeated until the number of repeated executions reaches the first preset number of times, but the task still fails to execute. If it is confirmed that the task cannot be executed normally, the name of the task that failed and the task location can be recorded to facilitate the user to check, clarify the specific task location of the failed execution, and analyze the reason for the task execution failure.

[0035] The beneficial effect of the above scheme is that it can perform multiple checks when a task fails, thereby verifying whether the task failure is an accidental event or a real failure, avoiding the problem that an accidental failure stops the task execution and causes the entire data daily settlement task to fail. In addition, when the number of failures reaches the first preset number, the failed task name and task location are recorded to facilitate further inspection and problem analysis.

[0036] In an embodiment of the present application, the method further includes:

[0037] Determine, according to the recorded failed task name and task location, a preset number of upstream tasks traced back to the failed task; wherein the upstream tasks are tasks that need to be executed before the failed task;

[0038] Starting from the preset number of upstream tasks, re-execute each task and determine whether the failed tasks are successfully executed;

[0039] If the execution is successful, the downstream tasks will continue to be executed;

[0040] If the execution fails and the number of repeated executions reaches a second preset number, the user is prompted.

[0041] Among them, the preset number can be determined according to the actual situation. Forward tracing means tracing forward. The number of times can be determined according to the actual situation. For example, after recording the name of the task that failed to execute and the task location, the preset number of upstream tasks that failed to execute can be determined. Assuming that the execution order of the tasks is Task A-Task B-Task C-Task D, the preset number is 2. If Task D fails to execute, the preset number of upstream tasks that can be traced back can be determined, that is, tracing back to the second upstream task from the task that failed to execute, that is, Task B. Thus, starting from the preset number of upstream tasks, re-execute the tasks, that is, execute each task from the beginning again, so as to detect whether the task that failed to execute last time still fails. If the task that failed to execute last time is executed successfully, continue to execute the downstream task. If the execution fails, repeat the failed task. If the number of repeated executions reaches the second preset number of times and all fail, remind the customer.

[0042] The beneficial effect of the above scheme is that when a task fails to execute, it can be re-executed from the upstream task to verify whether the task failed due to problems with the execution result of the upstream task. If so, it can be re-executed from the execution process of the upstream task to solve the failure problem.

[0043] The scheme in the embodiment of the present application obtains a task set list from a component database, parses the task set list, and determines each task and the command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and execution order in all steps of the data daily settlement task, and is generated according to the directed acyclic graph; the executability of the command line to be executed is detected, and according to the actual business scenario of the data daily settlement task, the tasks in the task set list are personalized and scheduled to determine the tasks to be executed serially and the tasks to be executed in parallel; when the command line to be executed is executable, for the tasks to be executed serially, the command lines for calling the execution program corresponding to each task are respectively executed in the order of serial execution, and the tasks are executed serially; for the tasks to be executed in parallel, the command lines for calling the execution program corresponding to each task are respectively executed, and the tasks are executed in parallel. The technical scheme of the embodiment of the present application can perform personalized scheduling of the corresponding tasks in the directed acyclic graph according to the specific business scenario of the data daily settlement task, thereby realizing flexible adjustment of the daily settlement order, and realizing the combination of serial processing and parallel processing, thereby improving the processing efficiency of the data daily settlement task.

[0044] Figure 2 The process of the task processing method provided in the embodiment of the present application Figure 2 .like Figure 2 As shown, the task processing method includes the following steps:

[0045] S210. For all steps of the data daily settlement task, the tasks in the steps are taken as nodes, and the execution order between the tasks is taken as the pointing relationship between the nodes to construct a directed acyclic graph.

[0046] Exemplarily, in the process of constructing a directed acyclic graph, all steps of a conventional daily data settlement task can be determined, and the tasks in the steps are used as nodes, and the execution order between tasks is used as the pointing relationship between nodes to construct a directed acyclic graph.

[0047] In the embodiment of the present application, for all steps of the data daily settlement task, the tasks in the steps are taken as nodes, and the execution order between the tasks is taken as the pointing relationship between the nodes to construct a directed acyclic graph, including:

[0048] Name the tasks corresponding to the steps according to the functions or operations of the steps of the data daily settlement task, and define each named task as a node;

[0049] According to the execution order between the tasks in the steps, the pointing relationship between the named nodes is determined to construct a directed acyclic graph.

[0050] Exemplarily, the tasks corresponding to each step of the data daily settlement task can be used as nodes, the functions or operations of the steps of the data daily settlement task can be determined, and the tasks corresponding to the steps can be named according to the functions or operations. For example, they can be named "import_to_database", "transform_data", etc. as nodes of a directed acyclic graph. Define the dependencies between each node, that is, some nodes must be started after other nodes are executed. For example, the value-added tax carryover node must be executed after the price-tax separation is completed. The above scheme can establish a directed acyclic graph according to the steps of conventional data daily settlement tasks, so that the directed acyclic graph can be reused in the subsequent process of executing data daily settlement tasks in specific business scenarios, and the directed acyclic graph can be adaptively adjusted directly to realize the execution of data daily settlement tasks in specific business scenarios.

[0051] In the embodiment of the present application, for all steps of the data daily settlement task, the tasks in the steps are taken as nodes, and the execution order between the tasks is taken as the pointing relationship between the nodes to construct a directed acyclic graph, including:

[0052] Set the business parameters and technical parameters for the task execution corresponding to each node; wherein the business parameters include at least one of the task name, execution frequency, number of retries, timeout period, and task processing identifier; the technical parameters include at least one of the database connection information, data call path, and task execution conditions.

[0053] In the embodiment of the present application, the business parameters and technical parameters of the task execution corresponding to each node can also be set so that the task execution is carried out in accordance with the corresponding parameters during the task execution process. The business parameters include at least one of the task name, execution frequency, number of retries, timeout time and task processing identification; the technical parameters include at least one of the database connection information, data call path and task execution conditions. Among them, the task name can be consistent with the task name generated when the node in the directed acyclic graph is established in the above embodiment. The execution frequency is used to indicate the frequency of the data daily settlement task execution, that is, the time between two data daily settlement execution processes. The number of retries is used to limit the number of retries allowed when the data daily settlement task fails to execute. The timeout time is used to limit the time allowed to delay before outputting the execution result when executing the task. The task processing identification is used to reflect the business scenario of the specific data daily settlement task, such as the specific subject such as the system, user, department, etc. that requests to execute the data daily settlement task. The database connection information includes information such as the port and IP address connected to the database, and the data call path is the path used when making data calls. The task execution condition is a prerequisite for the task before, for example, before executing task B, the task must be completed and an accurate execution result must be obtained. The above scheme can provide a basis and restrictions for the normal and smooth execution of the data daily settlement task, and improve the accuracy and controllability of the execution of the data daily settlement task.

[0054] S220, mapping each node in the directed acyclic graph to an execution program of the task, and generating a command line to be executed for calling each execution program according to the execution program mapped to each node and the pointing relationship between the nodes in the directed acyclic graph, and constructing a task set list.

[0055] Exemplarily, an execution program for executing each task is constructed, for example, including functions and commands used to execute the task. Each node in the directed acyclic graph can be mapped to the execution program of the task, that is, the path of the execution program that needs to be called when executing the task of each node is marked, so that the corresponding execution program can be found according to the path when the task needs to be executed. According to the execution program mapped by the node and the pointing relationship between the nodes in the directed acyclic graph, a command line to be executed for calling the execution program is generated, that is, according to the pointing relationship between the nodes, the execution order between the tasks corresponding to the nodes is determined, and then the calling order between the execution programs corresponding to the tasks is determined, and the command line to be executed for calling the execution program is generated according to the calling order, so as to execute the command line to be executed to call the execution program and realize the processing of the task.

[0056] In an embodiment of the present application, mapping each node in the directed acyclic graph to an execution program of a task includes:

[0057] Determine an execution program for executing tasks corresponding to each node; wherein the execution program includes functions and / or commands;

[0058] Determine a calling path for calling the execution program of the corresponding task when executing to each node, so as to map each node to the execution program of the task.

[0059] Exemplarily, for the task corresponding to each node, it is necessary to determine the execution program that needs to be executed to execute the task. The execution program includes functions and / commands, such as Python functions, Bash commands, Perl commands, SQL queries, etc. Determine the call path of the corresponding execution program that needs to be called when executing each node according to the pointing relationship between the nodes, so as to clarify which call path should be used to call the execution program for the task that needs to be executed on the node, so as to map each node to the execution program of the task. The beneficial effect of the above scheme is that the node can be mapped to the execution program of the task, so that the call path can be clarified when the task is executed subsequently, and the execution of the task can be realized by directly calling the execution program according to the call path.

[0060] In the embodiment of the present application, after determining the pointing relationship between the named nodes according to the execution order between the tasks in the steps, the method further includes:

[0061] For each subtask group in the data daily settlement task, determine the execution order between the subtask groups, and adjust the order between the node chains corresponding to each subtask group in the directed acyclic graph according to the execution order between the subtask groups;

[0062] The execution order of each task in the subtask group is determined, and the pointing relationship between each node in the node chain corresponding to the subtask group in the directed acyclic graph is adjusted according to the execution order of each task.

[0063] Among them, the tasks in a stage of the data daily settlement task can be used as a subtask group. Exemplarily, for each subtask group of the data daily settlement task, the execution order between each subtask group can be determined to ensure the logical correctness of the overall process. Organize the associated nodes into logical subtask groups. Each subtask group can be classified and organized according to function, data flow or business logic. Determine the execution order between different task groups, that is, which subtask groups should be executed first and which subtask groups can be executed in parallel. According to business needs and data dependencies, adjust the order between task groups to maximize the parallel execution and overall efficiency of tasks. According to the execution order between subtask groups, adjust the order between the node chains corresponding to each subtask group in the directed acyclic graph. The node chain is a chain obtained by connecting the nodes corresponding to the subtask group through a pointing relationship. Determine the execution order of each task in the subtask group, confirm the execution order and dependency of the tasks, and ensure that the tasks within each subtask group are executed in the correct order. For each subtask group, confirm the execution order between the tasks to ensure that the subsequent tasks are executed after the predecessor tasks are successfully completed. It is necessary to consider the situation where the task execution fails and set an appropriate retry strategy or error handling mechanism. According to the execution order of each task, the pointing relationship between the nodes in the node chain corresponding to the subtask group in the directed acyclic graph is adjusted, so as to further verify the pointing relationship between the nodes in the directed acyclic graph.

[0064] S230: Obtain a task set list from a component database, parse the task set list, and determine each task and a command line to be executed.

[0065] S240, detecting the executability of the command line to be executed, and performing personalized scheduling on the tasks in the task set list according to the actual business scenario of the data daily settlement task, and determining the tasks to be executed serially and the tasks to be executed in parallel.

[0066] In the embodiment of the present application, detecting the executability of the command line to be executed includes:

[0067] Distinguish between request scheduling commands and data processing transaction commands in the command lines to be executed;

[0068] Verify the legality of the name of the called execution program for request scheduling commands and data processing transaction commands;

[0069] For data processing transaction commands, check whether there are syntax problems and / or injection problems of abnormal query commands.

[0070] Exemplarily, the executability of the command line to be executed may include the legality of the naming of the command line to be executed, whether there are grammatical problems, whether there are security risks, and injection of abnormal query statements. In other words, it is to check whether the command line to be executed is executable in terms of its form. If the command line to be executed is not named in accordance with the prescribed naming method, or there is a problem with the syntax of the command line to be executed, it does not conform to the syntax that the programming language should have, or there are security risks, including commands that threaten data security, or there are abnormal query statements, for example, data A does not allow user uesr1 to query, but the command line to be executed contains a statement to query data A. By checking the executability of the command line to be executed, it is possible to check in advance whether the command line to be executed can be executed normally before the data daily settlement task is processed, so as to avoid the problem that the command line to be executed is not executable during the data daily settlement task processing, resulting in the failure of task execution, affecting the user experience.

[0071] In the embodiment of the present application, according to the actual business scenario of the data daily settlement task, the tasks in the task set list are personalized and scheduled to determine the tasks to be executed serially and the tasks to be executed in parallel, including:

[0072] Get the request task group corresponding to the business scenario of the actual data daily settlement task;

[0073] For each task in the request task group, determine a task to be executed serially and a task to be executed in parallel;

[0074] For each of the request task groups, a request task group to be processed serially and a request task group to be executed in parallel are determined.

[0075] For example, in the actual application process, for the business scenario of the actual data daily settlement task, the request task group corresponding to the data daily settlement task in the business scenario is obtained, and for each task in the request task group, the tasks to be executed in series and the tasks to be executed in parallel are determined, so as to make personalized adjustments to the directed acyclic graph according to the execution order of the tasks in the data daily settlement task in the actual business scenario. For each request task group, the request task group to be executed in series and the request task group to be executed in parallel are determined to clarify the execution order between each task group, so as to facilitate personalized adjustments to the directed acyclic graph.

[0076] S250, when the command line to be executed is executable, for the tasks to be executed serially, the command lines for calling the execution program corresponding to each task are respectively executed in the order of serial execution, and the tasks are executed serially;

[0077] S260: For the tasks to be executed in parallel, respectively execute the command lines of the calling execution program corresponding to each task, and execute the tasks in parallel.

[0078] In an embodiment of the present application, the method further includes:

[0079] If all tasks are completed, check whether each task is successfully executed;

[0080] If yes, a successful execution message is returned to prompt the user to perform subsequent operations.

[0081] For example, if all tasks are completed, it is detected whether each task is successfully executed and whether the correct execution results can be output. If yes, a successful execution message is returned to prompt the user to continue with subsequent operations, such as verifying the execution results of the data daily settlement task, so as to grasp the execution process of the entire data daily settlement task at a macro level and provide timely feedback.

[0082] In an embodiment of the present application, the method further includes:

[0083] If the data daily settlement task is completed and executed successfully, the execution result is verified according to the verification relationship between different types of data daily settlement tasks, the data balance verification relationship and the data summary relationship;

[0084] If the verification fails, the user will be notified, or the execution result will be handled according to the fallback principle.

[0085] Exemplarily, after the posting is completed, the accuracy of the execution results is checked based on the verification relationship between different types of data daily settlement tasks, the data balance verification relationship, and the data summary relationship, and feedback is given to the data that fails the verification, or it is processed according to the bottom-line rules.

[0086] The scheme in the embodiment of the present application, for all the steps of the data daily settlement task, takes the tasks in the steps as nodes, takes the execution order between the tasks as the pointing relationship between the nodes, and constructs a directed acyclic graph; maps each node in the directed acyclic graph to the execution program of the task, and generates the command line to be executed for calling each execution program according to the execution program mapped by each node and the pointing relationship between the nodes in the directed acyclic graph, and constructs a task set list. Obtain the task set list from the component database, parse the task set list, and determine each task and the command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and the execution order in all the steps of the data daily settlement task, and is generated according to the directed acyclic graph; detect the executability of the command line to be executed, and according to the actual business scenario of the data daily settlement task, perform personalized scheduling on the tasks in the task set list, and determine the tasks to be executed serially and the tasks to be executed in parallel; when the command line to be executed is executable, for the tasks to be executed serially, execute the command line of the calling execution program corresponding to each task in the order of serial execution respectively, and execute the tasks serially; for the tasks to be executed in parallel, execute the command line of the calling execution program corresponding to each task respectively, and execute the tasks in parallel. The technical solution of the embodiment of the present application can construct a directed acyclic graph based on conventional data daily settlement tasks, so that the directed acyclic graph can be reused. In the subsequent execution of data daily settlement tasks in specific business scenarios, it is only necessary to make adaptive adjustments to the directed acyclic graph, which solves the problem of low efficiency in data daily settlement task processing due to the serialization of the current daily settlement steps, and the problem that the fixed-line daily settlement processing sequence is difficult to adjust according to the actual business scenario, resulting in low applicability of the processing method, thereby improving the applicability of the processing method. It can also realize parallel processing between tasks, thereby improving the efficiency of data daily settlement task processing.

[0087] For example, Figure 3 As shown, the embodiment of the present application provides a specific implementation method of a task processing method, and the embodiment of the present application takes the daily account settlement task as an example. The specific steps are as follows:

[0088] (1) Directed Acyclic Graph (DAG) Drawing Module

[0089] Determine the daily closing steps and dependencies: All the daily closing steps involved need to be clearly defined and listed, and the specific tasks or operations that need to be performed in each step. The steps can be completed by initializing business parameters.

[0090] 1) Design DAG graph

[0091] Use the scheduling configuration platform to design a DAG graph, represent the daily steps as nodes in the DAG, and connect these nodes through directed edges (dependencies).

[0092] Node definition: Each daily step is a node in DAG. The node name can be named according to the specific function or operation of the task, such as "import_to_database", "transform_data", etc.

[0093] Edge (dependency) definition: Define the dependency between each node, that is, some nodes must be executed before other nodes can be started. For example, the VAT carry-over node must be executed after the price-tax separation is completed.

[0094] 2) Task definition

[0095] Define the specific task to be executed for each node, which can be a Python function, Bash command, Perl command, SQL query, or other executable task type.

[0096] 3) Parameter settings

[0097] Provide the required parameters for the task, including business parameters and technical parameters. Business parameters include at least one of the task name, execution frequency, number of retries, timeout period, and task processing identifier; technical parameters include at least one of the database connection information, data call path, and task execution condition. When configuring DAG, you need to consider scheduling strategies and parameters to ensure that daily tasks can be executed on time and accurately. Set the execution frequency of daily tasks, usually at a fixed time point every day or flexibly adjusted according to needs. Configure the number of retries and timeout period for the task to ensure that the task can automatically recover or alarm in abnormal situations.

[0098] (2) DAG parsing and verification module

[0099] In this step, the nodes and edges in the DAG need to be mapped to specific executable tasks and the dependencies between tasks.

[0100] 1) Node mapping as tasks

[0101] Query the full task parameter table and map each DAG node to the actual execution task, such as Python function, Bash command, SQL query, etc. Determine the completeness of the parameters and configuration data required for each task, such as whether the input data, output path, execution environment, etc. are legal. Determine the dependencies between tasks, that is, which tasks need to be started after other tasks are completed. Map the edges in the DAG to the dependencies between tasks in the scheduling script to ensure that tasks are executed in the correct order.

[0102] 2) Confirmation of the order between task groups

[0103] In this step, the execution order between task groups (node ​​groups) needs to be confirmed and adjusted to ensure the logical correctness of the overall process. Organize the associated nodes into logical task groups. Task groups can be classified and organized according to functions, data flows, or business logic. Determine the execution order between different task groups, that is, which task groups should be executed first and which task groups can be executed in parallel. Adjust the order between task groups based on business needs and data dependencies to maximize the parallel execution of tasks and overall efficiency.

[0104] 3) Confirmation of order within the task group

[0105] Within each task group, it is necessary to confirm the execution order and dependencies of the tasks to ensure that the tasks within each task group are executed in the correct order. For each task group, confirm the execution order between the tasks to ensure that the subsequent tasks are executed after the predecessor tasks are successfully completed. It is necessary to consider the situation where the task execution fails and set an appropriate retry strategy or error handling mechanism. This step ultimately generates a task set list with the sequence labels of the components within the group, which is persistently stored in the component database.

[0106] (3) Scheduling automatic generation module

[0107] The module performs the following steps: task list parsing, task sequence verification, parallel processing within request groups, serial processing between request groups, and returning a successful execution flag.

[0108] 1) Task set list analysis

[0109] Obtain the task set list in the component database, distinguish between request scheduling commands and data processing transaction commands, verify the legality of the script names involved in the commands, and check whether the data processing transaction commands have syntax problems, security risk injection, SQL injection, etc. Output the results of the inspection and correction to the next stage in the established format.

[0110] 2) Task sequence verification

[0111] Ensure that the execution order and dependencies of the tasks in the generated task list are in line with expectations to avoid task execution failure or data inconsistency due to incorrect order. Verify the dependencies of the generated task list to confirm the execution order between tasks. Check the dependencies within and between task groups to ensure that each task is executed under the correct conditions.

[0112] 3) Parallel processing within a request group

[0113] For tasks that can be processed in parallel within the same request group, perform reasonable parallel configuration to improve task execution efficiency and overall performance. Determine which tasks can be executed in parallel and set appropriate concurrency and resource limits. Use the parallel task management function provided by the scheduling platform to ensure that parallel execution within the task group does not affect each other.

[0114] 4) Serial processing between request groups

[0115] Configure the serial processing order between different request groups according to business logic or data dependencies to ensure that tasks are executed in the correct order. Determine which task groups need to be executed serially and set the order dependencies between task groups. Configure the dependencies of task groups in the scheduling platform to ensure that the subsequent task groups are executed only after the previous task groups are executed successfully.

[0116] 5) Return the execution success flag

[0117] After the task is completed, it is necessary to return a success or failure flag based on the task execution result for subsequent monitoring and error handling. Monitor and record the execution results of each task, and record the success or failure flag. According to the actual situation of task execution, update the task status and return the execution success flag for system monitoring and feedback.

[0118] 6) Accounting data verification module

[0119] After the posting is completed, the accuracy of the accounting data is checked based on the verification relationship between accounts, the balance verification relationship between debits and credits, and the summary relationship between institutions. Feedback is given to data that fails the verification, or it is processed according to the bottom-line rules.

[0120] Figure 4 This is a schematic diagram of the structure of the task processing device provided in the embodiment of the present application. Figure 4 As shown, the task processing device includes: a parsing module 310, which is used to obtain a task set list from a component database, parse the task set list, and determine each task and a command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and execution order in all steps of the data daily settlement task, and is generated according to the directed acyclic graph;

[0121] The inspection module 320 is used to detect the executability of the command line to be executed, and according to the actual business scenario of the data daily settlement task, perform personalized scheduling on the tasks in the task set list to determine the tasks to be executed serially and the tasks to be executed in parallel;

[0122] The serial execution module 330 is used to execute the command lines for calling the execution program corresponding to each task in the order of serial execution respectively, if the command line to be executed is executable, so as to execute the tasks serially;

[0123] The parallel execution module 340 is used to execute the command line of the calling execution program corresponding to each task for the parallel execution tasks, and execute the tasks in parallel.

[0124] In some embodiments, the process of determining the task set list includes:

[0125] For all the steps of the daily data settlement task, the tasks in the steps are taken as nodes, and the execution order between tasks is taken as the pointing relationship between nodes to construct a directed acyclic graph;

[0126] Each node in the directed acyclic graph is mapped to an execution program of the task, and according to the execution program mapped to each node and the pointing relationship between the nodes in the directed acyclic graph, a command line to be executed for calling each execution program is generated to construct a task set list.

[0127] In some embodiments, the device further includes a directed acyclic graph construction module, which is used to construct a directed acyclic graph for all steps of the data daily settlement task, using the tasks in the steps as nodes and the execution order between the tasks as the pointing relationship between the nodes, including:

[0128] Name the tasks corresponding to the steps according to the functions or operations of the steps of the data daily settlement task, and define each named task as a node;

[0129] According to the execution order between the tasks in the steps, the pointing relationship between the named nodes is determined to construct a directed acyclic graph.

[0130] In some embodiments, the directed acyclic graph construction module constructs a directed acyclic graph for all steps of the data daily settlement task, taking the tasks in the steps as nodes and the execution order between the tasks as the pointing relationship between the nodes, including:

[0131] Set the business parameters and technical parameters for the task execution corresponding to each node; wherein the business parameters include at least one of the task name, execution frequency, number of retries, timeout period, and task processing identifier; the technical parameters include at least one of the database connection information, data call path, and task execution conditions.

[0132] In some embodiments, the directed acyclic graph construction module maps each node in the directed acyclic graph to an execution program of the task, including:

[0133] Determine an execution program for executing tasks corresponding to each node; wherein the execution program includes functions and / or commands;

[0134] Determine a calling path for calling the execution program of the corresponding task when executing to each node, so as to map each node to the execution program of the task.

[0135] In some embodiments, after determining the pointing relationship between the named nodes according to the execution order between the tasks in the step, the apparatus further includes: an adjustment module, which is used to:

[0136] For each subtask group in the data daily settlement task, determine the execution order between the subtask groups, and adjust the order between the node chains corresponding to each subtask group in the directed acyclic graph according to the execution order between the subtask groups;

[0137] The execution order of each task in the subtask group is determined, and the pointing relationship between each node in the node chain corresponding to the subtask group in the directed acyclic graph is adjusted according to the execution order of each task.

[0138] In some embodiments, the verification module detects the executability of the command line to be executed, including:

[0139] Distinguish between request scheduling commands and data processing transaction commands in the command lines to be executed;

[0140] Verify the legality of the name of the called execution program for request scheduling commands and data processing transaction commands;

[0141] For data processing transaction commands, check whether there are syntax problems and / or injection problems of abnormal query commands.

[0142] In some embodiments, the inspection module 320 performs personalized scheduling on the tasks in the task set list according to the actual business scenario of the data daily settlement task, and determines the tasks to be executed serially and the tasks to be executed in parallel, including:

[0143] Get the request task group corresponding to the business scenario of the actual data daily settlement task;

[0144] For each task in the request task group, determine a task to be executed serially and a task to be executed in parallel;

[0145] For each of the request task groups, a request task group to be processed serially and a request task group to be executed in parallel are determined.

[0146] In some embodiments, the apparatus further comprises: a repeating execution module, configured to:

[0147] If there is a task execution failure, repeat the task;

[0148] If the number of times the task is repeatedly executed reaches a first preset number, the name and location of the task that failed to execute are recorded.

[0149] In some embodiments, the apparatus further comprises: an upstream node execution module, configured to:

[0150] Determine, according to the recorded failed task name and task location, a preset number of upstream tasks traced back to the failed task; wherein the upstream tasks are tasks that need to be executed before the failed task;

[0151] Starting from the preset number of upstream tasks, re-execute each task and determine whether the failed tasks are successfully executed;

[0152] If the execution is successful, the downstream tasks will continue to be executed;

[0153] If the execution fails and the number of repeated executions reaches a second preset number, the user is prompted.

[0154] In some embodiments, the device further comprises: a success prompt module, configured to:

[0155] If all tasks are completed, check whether each task is successfully executed;

[0156] If yes, a successful execution message is returned to prompt the user to perform subsequent operations.

[0157] In some embodiments, the device further comprises: a verification module, configured to:

[0158] If the data daily settlement task is completed and executed successfully, the execution result is verified according to the verification relationship between different types of data daily settlement tasks, the data balance verification relationship and the data summary relationship;

[0159] If the verification fails, the user will be notified, or the execution result will be handled according to the fallback principle.

[0160] The task processing device provided in the embodiment of the present application can be used to execute the technical solution of the task processing method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0161] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the target transfer data determination module 410 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above target transfer data determination module 410. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0162] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0163] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes the scheme in the above embodiment. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0164] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0165] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0166] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0167] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0168] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the task processing method in the above embodiment.

[0169] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the task processing method of the above embodiment.

[0170] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, the technical solution of the task processing method in the above embodiment can be implemented.

[0171] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0172] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A task processing method, characterized in that: include: Obtaining a task set list from a component database, parsing the task set list, and determining each task and a command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and execution order in all steps of the data daily settlement task, and is generated according to the directed acyclic graph; Detect the executability of the command line to be executed, and according to the actual business scenario of the data daily settlement task, perform personalized scheduling on the tasks in the task set list to determine the tasks to be executed serially and the tasks to be executed in parallel; When the command line to be executed is executable, for the tasks to be executed serially, the command lines for calling the execution program corresponding to each task are executed in the order of serial execution, and the tasks are executed serially; For tasks to be executed in parallel, the command lines for calling execution programs corresponding to each task are executed respectively to execute the tasks in parallel.

2. The method according to claim 1, characterized in that The process of determining the task set list includes: For all the steps of the daily data settlement task, the tasks in the steps are taken as nodes, and the execution order between tasks is taken as the pointing relationship between nodes to construct a directed acyclic graph; Each node in the directed acyclic graph is mapped to an execution program of the task, and according to the execution program mapped to each node and the pointing relationship between the nodes in the directed acyclic graph, a command line to be executed for calling each execution program is generated to construct a task set list.

3. The method according to claim 2, characterized in that For all the steps of the daily data settlement task, the tasks in the steps are taken as nodes, and the execution order between tasks is taken as the pointing relationship between nodes to construct a directed acyclic graph, including: Name the tasks corresponding to the steps according to the functions or operations of the steps of the data daily settlement task, and define each named task as a node; According to the execution order between the tasks in the steps, the pointing relationship between the named nodes is determined to construct a directed acyclic graph.

4. The method according to claim 2, characterized in that: For all the steps of the daily data settlement task, the tasks in the steps are taken as nodes, and the execution order between tasks is taken as the pointing relationship between nodes to construct a directed acyclic graph, including: Set the business parameters and technical parameters for the task execution corresponding to each node; wherein the business parameters include at least one of the task name, execution frequency, number of retries, timeout period, and task processing identifier; the technical parameters include at least one of the database connection information, data call path, and task execution conditions.

5. The method according to claim 2, characterized in that: Mapping each node in the directed acyclic graph to an execution program of the task includes: Determine an execution program for executing tasks corresponding to each node; wherein the execution program includes functions and / or commands; Determine a calling path for calling the execution program of the corresponding task when executing to each node, so as to map each node to the execution program of the task.

6. The method according to claim 2, characterized in that After determining the pointing relationship between the named nodes according to the execution order between the tasks in the steps, the method further includes: For each subtask group in the data daily settlement task, determine the execution order between the subtask groups, and adjust the order between the node chains corresponding to each subtask group in the directed acyclic graph according to the execution order between the subtask groups; The execution order of each task in the subtask group is determined, and the pointing relationship between each node in the node chain corresponding to the subtask group in the directed acyclic graph is adjusted according to the execution order of each task.

7. The method according to claim 1, characterized in that Check the executability of the command line to be executed, including: Distinguish between request scheduling commands and data processing transaction commands in the command lines to be executed; Verify the legality of the name of the called execution program for request scheduling commands and data processing transaction commands; For data processing transaction commands, check whether there are syntax problems and / or injection problems of abnormal query commands.

8. The method according to claim 1, characterized in that: According to the actual business scenario of the data daily settlement task, the tasks in the task set list are personalized and scheduled to determine the tasks to be executed serially and in parallel, including: Get the request task group corresponding to the business scenario of the actual data daily settlement task; For each task in the request task group, determine a task to be executed serially and a task to be executed in parallel; For each of the request task groups, a request task group to be processed serially and a request task group to be executed in parallel are determined.

9. The method according to claim 1, characterized in that: The method further comprises: If there is a task execution failure, repeat the task; If the number of times the task is repeatedly executed reaches a first preset number, the name and location of the task that failed to execute are recorded.

10. The method according to claim 9, characterized in that The method further comprises: Determine, according to the recorded failed task name and task location, a preset number of upstream tasks traced back to the failed task; wherein the upstream tasks are tasks that need to be executed before the failed task; Starting from the preset number of upstream tasks, re-execute each task and determine whether the failed tasks are successfully executed; If the execution is successful, the downstream tasks will continue to be executed; If the execution fails and the number of repeated executions reaches a second preset number, the user is prompted.

11. The method according to claim 1, characterized in that: The method further comprises: If all tasks are completed, check whether each task is successfully executed; If yes, a successful execution message is returned to prompt the user to perform subsequent operations.

12. The method according to claim 11, characterized in that The method further comprises: If the data daily settlement task is completed and executed successfully, the execution result is verified according to the verification relationship between different types of data daily settlement tasks, the data balance verification relationship and the data summary relationship; If the verification fails, the user will be notified, or the execution result will be handled according to the fallback principle.

13. A task processing device, characterized in that: include: A parsing module is used to obtain a task set list from a component database, parse the task set list, and determine each task and a command line to be executed; wherein the task set list is a directed acyclic graph constructed according to the tasks and execution order in all steps of the data daily settlement task, and is generated according to the directed acyclic graph; The inspection module is used to detect the executability of the command line to be executed, and according to the actual business scenario of the data daily settlement task, perform personalized scheduling on the tasks in the task set list to determine the tasks to be executed serially and the tasks to be executed in parallel; A serial execution module is used to execute the command lines of the calling execution program corresponding to each task in the order of serial execution when the command line to be executed is executable, so as to execute the tasks serially; The parallel execution module is used to execute the command line of the calling execution program corresponding to each task for the parallel execution tasks, so as to execute the tasks in parallel.

14. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the task processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the task processing method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the task processing method according to any one of claims 1 to 12.

Images