An implementation method based on a functional definition process

Abstract

An implementation method based on a functional definition process provided by the present invention defines flexible process-level parameters, which are divided into four types: input parameters, output parameters, input / output parameters, and internal parameters. Through a parameter exchange area, information transfer between processes, between task operators, and between processes and operators is achieved. And during the transfer process, the process parameters can be changed into new values at any time to meet the calculation requirements of new task operators. The selection of multiple parameter exchange areas can choose memory, database tables, or other storage to meet different deployment scenarios. User-level parameters are defined as calculation constants, which are only allowed to be referenced and not changed during the calculations of all processes and task nodes. The parameters of task operators reference fixed values, user-level parameters, or process-level parameters. After the task is executed, the process-level parameters are changed. It solves the problems of information and data transfer between processes, between task operators, and between processes and operators.

Description

Technical Field

[0001] The present invention relates to the field of distributed task scheduling systems, and in particular to an implementation method based on a functional definition process. Background Art

[0002] Distributed scheduling systems, such as the open source systems Azkaban and Oozie, are task scheduling systems that integrate process orchestration, task scheduling, task execution, and task monitoring. They have the following features:

[0003] Visual process arrangement page: Based on the DSL design mode, tasks can be arranged in serial or parallel processes by dragging and dropping to form a directed acyclic graph of the task process.

[0004] Distributed deployment and operation: The system is generally composed of components such as Master, Worker, and ApiServer. It can be deployed and operated on a single machine or in a cluster; each service component can be dynamically expanded. ApiServer is responsible for interacting with the front end and providing Api services, Master is responsible for task scheduling, and Worker is responsible for specific task execution.

[0005] Multiple process execution modes: single-step execution, overall execution, and scheduled execution; process execution can be stopped, paused, and resumed; failed processes can be resumed, retried, skipped, etc. The running status of tasks and processes can be monitored.

[0006] Task operators: A variety of task operators meet different business scenarios. Monitoring pages: Such as dashboards, Gantt charts, attribute charts, etc., to meet different monitoring needs.

[0007] The distributed task scheduling system can break down a complex business process into several independent subtasks, which are scheduled by the Master and distributed to each Worker for execution. The system simplifies a lot of work for users and improves work efficiency. However, there is still an important problem with the current distributed task scheduling system: information and data cannot be dynamically modified and transferred within or between processes. During the process running, tasks are relatively independent of each other, and parameters can only flow within operators. There is a lack of interaction between operators. Operators that have been executed cannot pass the calculated results to the process or other task operators, which cannot meet some scenarios where the later executed tasks depend on the results of the previous task execution; at the same time, parameter transfer cannot be achieved between processes, and existing business processes cannot be reused to build complex business scenarios. Summary of the invention

[0008] In view of the above problems, the present invention is proposed to provide an implementation method based on a functional definition process that overcomes the above problems or at least partially solves the above problems.

[0009] According to one aspect of the present invention, there is provided an implementation method based on a functional definition process, and the implementation method includes:

[0010] Define user-level parameters, and perform initialization processing on the user-level parameters to obtain initialized user-level parameters;

[0011] Define a process and process-level parameters, and perform initialization processing on the process-level parameters to obtain initialized process-level parameters;

[0012] Define task operators and task-level parameters in the process, and set the initialized task-level parameters;

[0013] Pass the process-level parameters through a parameter exchange area to achieve information transfer between processes, between task operators, and between processes and operators;

[0014] During the transfer process, change the process parameters to new values to obtain updated process parameters;

[0015] During the transfer process, change the user-level parameters to new values to obtain updated user-level parameters;

[0016] Save the updated process parameters and the updated user-level parameters to the parameter exchange area for use by task operators;

[0017] The task operator starts to execute, and determines whether the task-level parameters reference the user-level parameters and the process-level parameters. If so, the task operator obtains new parameter values from the parameter exchange area; otherwise, it uses the initialization values;

[0018] After the task operator finishes executing, determine whether the task-level parameters reference the process-level parameters. If so, replace the values of the process-level parameters; otherwise, update the new process-level parameter values to the parameter exchange area for use by subsequent task grandchildren.

[0019] Optionally, the storage space specifically includes: memory, database tables, and storage devices.

[0020] Optionally, the process-level parameters specifically include four types: input parameters, output parameters, input / output parameters, and internal parameters.

[0021] Optionally, the defined user-level parameters are constants and are read-only during the process execution and cannot be changed.

[0022] Optionally, the task operator parameters include three types: input parameters, output parameters, and input / output parameters.

[0023] Optionally, when the process starts running, all initialization values are saved to the parameter exchange area, and the parameter exchange area includes memory, database tables, and external storage devices.

[0024] Optionally, when the task operator executes, it references the user-level parameters and the process-level parameters, and obtains parameter values from the parameter exchange area.

[0025] After the task operator finishes execution, it dynamically replaces the process-level parameters to obtain a replacement result.

[0026] The replacement result is updated to the parameter exchange area to achieve parameter replacement and parameter transfer between the task operator and the process, and between task operators.

[0027] The present invention provides process-level parameters with flexible definitions, which are divided into four types: input parameters, output parameters, input / output parameters, and internal parameters. Through the parameter exchange area, information transfer between processes, between task operators, and between processes and operators is achieved. And during the transfer process, the process parameters can be changed into new values at any time to meet the calculation requirements of new task operators. The selection of multiple parameter exchange areas allows selection of memory, database tables, or other storage to meet different deployment scenarios. User-level parameters are defined as calculation constants, which are only allowed to be referenced and not changed during the calculations of all processes and task nodes. The parameters of the task operator reference fixed values, user-level parameters, or process-level parameters. The process-level parameters are changed after the task execution is completed. This solves the problems of information and data transfer between processes, between task operators, and between processes and operators.

[0028] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the present invention more obvious and understandable, the following specifically illustrates the specific embodiments of the present invention. Brief Description of the Drawings

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0030] Figure 1 A method for implementing a process defined based on functional definition provided for an embodiment of the present invention. Detailed Embodiments

[0031] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0032] In the embodiments of the specification, claims, and drawings of the present invention, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a series of steps or units are included.

[0033] The following further describes the technical solutions of the present invention in detail with reference to the drawings and embodiments.

[0034] Define user-level parameters as constants, which are read-only during the process execution and cannot be changed. For example: USER_A = 1, USER_B = 2, and user-level parameters are referenced in subsequent task operators.

[0035] Define process-level parameters and initial values, similar to the input and output parameter definitions of a function. The input parameter of the function is a value reference or an address reference, and the output parameter is the function return value. The process parameters are divided into four types: input parameter IN, output parameter OUT, input / output parameter INOUT, and internal parameter INSIDE. For example:

[0036] {"param": "process_A1", "type": "IN", "value": 1}

[0037] {"param": "process_A2", "type": "INSIDE", "value": 2}

[0038] {"param": "process_A3", "type": "OUT", "value": 3}

[0039] {"param": "process_A4", "type": "INOUT", "value": 4}

[0040] During the process operation, all variable values are referenced and changed by the task operators. The difference is that when the process starts, only the IN and INOUT parameters are allowed to be passed in and modified; when the process ends, only the OUT and INOUT parameters are allowed to be passed to the parent process, and the INSIDE parameter can neither be passed in at startup nor passed out at the end.

[0041] Define task-level parameters and initial values. The parameters are divided into three types: input parameter IN, output parameter OUT, and input / output parameter INOUT. They are defined as fixed values and reference user-level parameters or process-level parameters. For example:

[0042] {"param": "task_A1", "type": "IN", "value": 1}

[0043] {"param": "task_A2", "type": "IN", "value": ${USER_A}}

[0044] {"param": "task_A3", "type": "OUT", "value": ${process_A3}}

[0045] {"param": "task_A4", "type": "INOUT", "value": ${process_A4}}

[0046] task_A1 defines a fixed value, and task_A2, task_A3, and task_A4 respectively reference user-level parameters and process-level parameters.

[0047] When starting the process, user-level parameters and process-level parameters are passed in. The user-level parameters are automatically passed in and invisible to the user. For process-level IN or INOUT type parameters, the user can modify the new value on the start page to replace the defined default value. For example:

[0048] process_A1 = 10 (the default value is 1, changed to 10)

[0049] process_A4 = 40 (the default value is 4, changed to 40)

[0050] Since process_A2 and process_A3 are of INSIDE and OUT types respectively, the defined default values cannot be replaced when starting the process.

[0051] When the task operator executes, the task-level parameters can reference the values of user-level parameters and process-level parameters. For example, the above task operator parameters task_A2, task_A3, and task_A4 respectively reference user-level parameters and process-level parameters, and these parameters respectively obtain new parameter values from the parameter exchange area during runtime.

[0052] At the same time, in the execution script of the task operator itself, it can also reference its own parameters, or reference process-level parameters or user-level parameters. For example, the request script of the HTTP operator:

[0053] http: / / 10.0.34.115:8080 / fyi?P1=${task_A1}&P2={process_A1}&P3=${USER_A}

[0054] During runtime, the parameters in the script are also replaced with actual values (the implementation logic inside the task operator).

[0055] When the task operator finishes execution, if the task-level parameter "refers to" a process-level parameter, it will replace the value of the process-level parameter and update the new process-level parameter value to the parameter exchange area. This "replacement" function requires that the task parameter type must be OUT or INOUT, and when the parameter is defined, the parameter should refer to the parameter of the process. As the task-level parameters defined above,

[0056] task_A3, with the type of OUT, refers to the process parameter process_A3

[0057] task_A4, with the type of INOUT, refers to the process parameter process_A4

[0058] Both of these parameters have the ability to replace process parameters, while task_A1 and task_A2 do not have the ability to replace process parameters. When the task operator ends, the operator will replace the process parameters internally and update the replaced process parameter values to the parameter exchange area.

[0059] In summary, for the parameters of the task operator with the parameter replacement function, they must: be of the type OUT or INOUT; refer to the process parameters.

[0060] The parameter transfer between processes is achieved through sub-process operators. For a sub-process to change the parameters of the parent process, the parameters of the sub-process must: be of the type OUT or INOUT; refer to the parameters of the parent process.

[0061] Take Figure 1 as an example, the process of process scheduling and parameter replacement is as follows:

[0062] The scheduling service starts a process and saves the initial values of the user parameters and process parameters to the parameter exchange area;

[0063] The scheduling service sends task operator one to an independent Worker (JVM) for execution. When task operator one runs, it obtains the values of the user parameters and process parameters from the parameter exchange area; when the operator ends, it updates the replaced process parameter values to the parameter exchange area;

[0064] The scheduling service schedules the next task operator for execution: the sub-process operator;

[0065] The sub-process starts. Similarly, the scheduling service sends task operator two in the sub-process to an independent Worker (JVM) for execution;

[0066] When task operator two starts, it obtains the user parameter value and the updated process parameter value from the parameter exchange area;

[0067] When task operator two ends, it updates the replaced process parameter values to the parameter exchange area again;

[0068] The scheduling service schedules the execution of the next task operator: Task Operator Three;

[0069] When Task Operator Three starts, it obtains the user parameter values and the updated process parameter values from the parameter exchange area; when Task Operator Three ends, since there are no more task operators, the scheduling service ends the entire process.

[0070] Regarding the parameter exchange area

[0071] After the task operator changes the parameter values of the process, it saves and updates them to the parameter exchange area. The parameter exchange area can choose memory, database tables, or other external storage. When subsequent task operators refer to the parameters of the process, they obtain the updated values from the parameter exchange area, thus realizing parameter replacement and transfer between task operators and processes, between task operators, and between processes.

[0072] When the parameter values are saved to database tables or other storage systems, the task operator can save and read them conveniently. When saved to memory, because it is a distributed system, it cannot be saved to its own Worker host. Instead, it needs to send a network request to save the changes to the memory cache of the Master host; when a new value is needed, it obtains it from the memory cache of the Master host through a network request.

[0073] If the parameter is saved to memory, once the Master fails, the data cannot be reliably restored. Generally, this method is not recommended for use in a production environment. For the database table method, because of the frequent access to the database table, it can adapt to the concurrent access of a small number of processes. For the concurrent access of a large number of processes, it is recommended to use Redis as the external storage, which has the characteristics of high efficiency and reliability.

[0074] Beneficial effects: To solve the problems of information and data transfer between processes, between task operators, and between processes and operators, we provide a solution that draws on the idea of function definition and defines the process in a functional way.

[0075] Define flexible process-level parameters, which are divided into four types: input parameter IN, output parameter OUT, input / output parameter INOUT, and internal parameter INSIDE. Through the parameter exchange area, realize information transfer between processes, between task operators, and between processes and operators, and during the transfer process, change the process parameters into new values at any time to meet the calculation requirements of new task operators.

[0076] Multiple choices for the parameter exchange area, which can choose memory, database tables, or other storage to meet different deployment scenarios.

[0077] Define user-level parameters as calculation constants, which are only allowed to be referenced and not changed during the calculations of all processes and task nodes.

[0078] The parameters of the task operator can reference fixed values, user-level parameters, or process-level parameters. The process-level parameters can be changed after the task is executed.

[0079] The above specific implementation manners have further elaborated on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific implementation manners of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for implementing a process based on functional definition, characterized in that The implementation method includes: Defining user-level parameters and performing initialization processing on the user-level parameters to obtain initialized user-level parameters. Among them, the user-level parameters are defined as calculation constants and are only allowed to be referenced but not changed in the calculations of all processes and task nodes; Defining processes and process-level parameters and performing initialization processing on the process-level parameters to obtain initialized process-level parameters; among them, the process-level parameters are divided into four types: input parameter IN, output parameter OUT, input / output parameter INOUT, and internal parameter INSIDE; and when the process starts running, all initialized values are saved to the parameter exchange area; Defining task operators and task-level parameters in the process and setting the initialized task-level parameters; When the process starts running, saving all initialized values to the parameter exchange area; Passing the process-level parameters through the parameter exchange area to achieve information transfer between processes, between task operators, and between processes and operators, including: the task operator references the user-level parameters and the process-level parameters when executing; obtaining parameter values from the parameter exchange area; after the task operator finishes execution, dynamically replacing the process-level parameters to obtain a replacement result; updating the replacement result to the parameter exchange area to achieve parameter replacement and parameter transfer between the task operator and the process and between task operators; Specifically including: when the task operator starts to execute, determining whether the task-level parameters reference the user-level parameters and the process-level parameters. If so, the task operator goes to the parameter exchange area to obtain the referenced parameter values. Otherwise, the initialized values are used; After the task operator finishes execution, determining whether the task-level parameters reference the process-level parameters. If so, replacing the values of the process-level parameters and updating the new process-level parameter values to the parameter exchange area for subsequent task operators to continue using; among them, the replacement function requires that the task-level parameter type must be OUT or INOUT type, and the parameter must reference the process-level parameter when the task-level parameter is defined; among them, the parameter exchange area specifically includes: memory, database table, and storage; 2. The implementation method of a functional definition-based process according to claim 1, characterized in that The defined user-level parameters are constants and are read-only and cannot be changed during the process execution; 3. The implementation method of a functional definition-based process according to claim 1, characterized in that The task-level parameters include three types: input parameter, output parameter, and input / output parameter.

Images