Templated assembly line component design method and system

Abstract

The invention relates to a templated assembly line component design method and system.The method comprises the steps that component template information configured by a user for each task in a visual interface is analyzed according to template specifications, and corresponding component parameter information is obtained; rendering the components on the assembly line arrangement interface according to the component parameter information to obtain the corresponding components and configuration files of the components, receiving behavior operation performed on the components by a user in the assembly line arrangement interface, generating an assembly line according to the behavior operation, and running the configuration files of the components in the assembly line through a task scheduler. And the execution of the assembly line is realized. Therefore, a user is supported to carry out task configuration and task arrangement on a visual interface in a template mode, the operation is simplified, the technical threshold is reduced, the task configuration and arrangement efficiency and the execution efficiency of an assembly line are improved, component template information is analyzed through a unified template specification, a unified specification system is adopted, and the operation stability is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a templated pipeline component design method and system. Background Art

[0002] With the development of data science and machine learning technology, pipeline management plays a vital role in handling complex tasks and workflows. Pipeline tasks usually involve data preprocessing, model training, model evaluation, and model deployment. Each link requires different tasks and tools, which makes the task scheduling and management complicated. In order to solve this problem, many systems and frameworks have emerged. The current mainstream pipeline management tools are: KubeflowPipelines, MLflow, Airflow, and Argo Workflows. Although these pipeline management tools have improved the efficiency of task scheduling and management to a certain extent, they all have the following defects:

[0003] (1) Complex configuration: Users need to define tasks through configuration files, which has high technical barriers;

[0004] (2) Difficulty in collaboration: The lack of unified parameter specifications and data transmission standards leads to complex data flow and dependency management between components, which is unstable and prone to errors.

[0005] (3) Inconvenient operation: Task management and design are performed using command lines or configuration files. It is impossible to intuitively view the dependencies and status of tasks, which affects the efficiency of task scheduling and management. Summary of the invention

[0006] The technical problem to be solved by the present invention is: the present invention provides a templated pipeline component design method and system, which simplifies the complexity of task scheduling and configuration, lowers the technical threshold, improves operational convenience, improves collaboration efficiency, and improves pipeline execution efficiency.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] In a first aspect, the present invention provides a templated pipeline component design method, comprising:

[0009] Receive the component template information configured by the user for each task in the visual interface, parse the component template information according to the template specification, obtain the corresponding component parameter information, render the component in the pipeline orchestration interface according to the component parameter information, obtain the corresponding component, and generate the configuration file of the component at the same time;

[0010] Receive the user's behavioral operation on the component in the pipeline orchestration interface, generate a pipeline according to the behavioral operation, and run the configuration file of the component in the pipeline through the task scheduler to implement the execution of the pipeline.

[0011] The beneficial effects of the present invention are: supporting users to configure tasks in a visual interface in a templated manner, without the need to write complex configuration files, lowering technical barriers, and improving operational convenience; and parsing component template information with a unified template specification to generate corresponding components and component configuration files. The unified specification system makes the collaboration between tasks smoother, the data transmission more reliable, improves the stability of operation, and reduces the error rate. In addition, a visual pipeline orchestration interface is provided for users to perform behavioral operations when performing task orchestration, which simplifies the complexity of task orchestration and improves the efficiency of task orchestration and the execution efficiency of the pipeline.

[0012] Optionally, before receiving the component template information configured by the user for each task in the visual interface, the process includes:

[0013] Receive the component type selected by the user for each task in the visual interface, determine whether the component type exists in the original component library, and if not, send the component template information to be created to the user, where the component template information to be created is used by the user to create a new component;

[0014] Receive the component template information to be created configured by the user for creating a new component, and generate configuration information to be confirmed in a preview form. When the configuration information to be confirmed is correct, send the configuration information to be confirmed to the original component library to realize the creation of the new component.

[0015] According to the above description, it can be seen that user-defined extension components are supported, and users only need to configure the template information of the component to be created, without modifying the core code, to complete the creation of new components, thereby improving the scalability and flexibility of pipeline design, and before sending the configuration information to be confirmed to the original component library to realize the creation of the new component, the configuration information to be confirmed will be generated in a preview form, that is, a visual preview confirmation mechanism is provided to improve the accuracy of new component creation and optimize the user experience.

[0016] Optionally, the template specification includes a first specification, a second specification and a third specification, and parsing the component template information according to the template specification includes:

[0017] Acquire input parameters, output parameters, control parameters, basic task information and task parameters from the component template information;

[0018] Parsing the input parameter according to the first specification to obtain an input type of the input parameter, and obtaining a component dynamic behavior according to the input type;

[0019] Parsing the output parameter according to the second specification to obtain an output type, and obtaining a component output product according to the output type;

[0020] Parsing the control parameter according to the third specification to obtain parameter information of the control parameter, and obtaining component execution behavior according to the parameter information;

[0021] The component dynamic behavior, the component output product, the component execution behavior, the task basic information and the task parameter are used as component parameter information.

[0022] According to the above description, a unified specification system is used to parse input parameters, output parameters and control parameters, and to standardize component specification behaviors, component output products and component execution behaviors, so that all components can interact according to unified rules, reduce errors and debugging difficulties caused by incompatible parameters, and ensure the stability of pipeline execution.

[0023] Optionally, the component types include: a data preprocessing component, a model training component, a model evaluation component, a model deployment component, a data visualization component and a notification component.

[0024] According to the above description, the diversification of component types improves the flexibility and reusability of the pipeline.

[0025] Optionally, the behavior operation includes dragging components and connecting components.

[0026] According to the above description, users can orchestrate pipelines by dragging components and connecting components in the pipeline orchestration interface, simplifying the complexity of pipeline orchestration and achieving rapid deployment.

[0027] In a second aspect, the present invention provides a templated pipeline component design system, comprising:

[0028] The task configuration module is used to receive the component template information configured by the user for each task in the visual interface, parse the component template information according to the template specification, obtain the corresponding component parameter information, render the component in the pipeline orchestration interface according to the component parameter information, obtain the corresponding component, and generate the configuration file of the component at the same time;

[0029] The task scheduling module is used to receive the user's behavioral operations on the components in the pipeline scheduling interface, generate a pipeline according to the behavioral operations, and run the configuration files of the components in the pipeline through the task scheduler to implement the execution of the pipeline.

[0030] The beneficial effects of the present invention are: supporting users to configure tasks in a visual interface in a templated manner, without the need to write complex configuration files, lowering technical barriers, and improving operational convenience; and parsing component template information with a unified template specification to generate corresponding components and component configuration files. The unified specification system makes the collaboration between tasks smoother, the data transmission more reliable, improves the stability of operation, and reduces the error rate. In addition, a visual pipeline orchestration interface is provided for users to perform behavioral operations when performing task orchestration, which simplifies the complexity of task orchestration and improves the efficiency of task orchestration and the execution efficiency of the pipeline.

[0031] Optionally, the task configuration module includes:

[0032] The component creation module is used to receive the component type selected by the user for each task in the visual interface, and determine whether the component type exists in the original component library. If not, the component template information to be created is sent to the user, and the component template information to be created is used by the user to create a new component;

[0033] Receive the component template information to be created configured by the user for creating a new component, and generate configuration information to be confirmed in a preview form. When the configuration information to be confirmed is correct, send the configuration information to be confirmed to the original component library to realize the creation of the new component.

[0034] According to the above description, it can be seen that user-defined extension components are supported, and users only need to configure the template information of the component to be created, without modifying the core code, to complete the creation of new components, thereby improving the scalability and flexibility of pipeline design, and before sending the configuration information to be confirmed to the original component library to realize the creation of the new component, the configuration information to be confirmed will be generated in a preview form, that is, a visual preview confirmation mechanism is provided to improve the accuracy of new component creation and optimize the user experience.

[0035] Optionally, the task configuration module is specifically:

[0036] The template specification includes a first specification, a second specification and a third specification, and parsing the component template information according to the template specification includes:

[0037] Acquire input parameters, output parameters, control parameters, basic task information and task parameters from the component template information;

[0038] Parsing the input parameter according to the first specification to obtain an input type of the input parameter, and obtaining a component dynamic behavior according to the input type;

[0039] Parsing the output parameter according to the second specification to obtain an output type, and obtaining a component output product according to the output type;

[0040] Parsing the control parameter according to the third specification to obtain parameter information of the control parameter, and obtaining component execution behavior according to the parameter information;

[0041] The component dynamic behavior, the component output product, the component execution behavior, the task basic information and the task parameter are used as component parameter information.

[0042] According to the above description, a unified specification system is used to parse input parameters, output parameters and control parameters, and to standardize component specification behaviors, component output products and component execution behaviors, so that all components can interact according to unified rules, reduce errors and debugging difficulties caused by incompatible parameters, and ensure the stability of pipeline execution.

[0043] Optionally, the component types include: a data preprocessing component, a model training component, a model evaluation component, a model deployment component, a data visualization component and a notification component.

[0044] According to the above description, the diversification of component types improves the flexibility and reusability of the pipeline.

[0045] Optionally, the behavior operation includes dragging components and connecting components.

[0046] According to the above description, users can orchestrate pipelines by dragging components and connecting components in the pipeline orchestration interface, simplifying the complexity of pipeline orchestration and achieving rapid deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of a templated pipeline component design method provided in this embodiment;

[0048] Figure 2 A schematic diagram of the overall process of a templated pipeline component design method provided in this embodiment;

[0049] Figure 3 A schematic diagram of the structure of a templated pipeline component design system provided in this embodiment.

[0050] [Description of Reference Numerals]

[0051] 1. A templated pipeline component design system;

[0052] 2. Task configuration module; 21. Component creation module;

[0053] 3. Task scheduling module. DETAILED DESCRIPTION

[0054] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0055] Embodiment 1

[0056] Please refer to Figure 1 to Figure 2 The present invention provides a templated pipeline component design method, comprising the steps of:

[0057] S1. Receive component template information configured by the user for each task in the visualization interface, parse the component template information according to the template specification, obtain corresponding component parameter information, render the component in the pipeline orchestration interface according to the component parameter information, obtain the corresponding component, and generate a configuration file for the component at the same time;

[0058] In this embodiment, the original component library is pre-built, and each component in the original component library has independent functions, and interactive operations are realized through standardized interfaces. All components have a unified component template structure, as shown in Table 1. In Table 1, the basic task information includes component ID and component name; task parameters include image, working directory, startup command, resource specification and mount path, which are used to configure the core options of the execution environment; input parameters are used to define the variables and data sources required for the operation of each component. The data set, code configuration and model in the input parameters can be used as introduction parameters to use the resources relied on by the platform for the task configuration of the pipeline; control parameters include environment variables, timeout interrupts and retries, which are used to perform fine-grained control and management of the execution of pipeline tasks; output parameters include output path and output variables, which are used to provide necessary input for subsequent components or to record task results, output variables are used for variable transfer, and output path is used for file product transfer. The basic task information and task parameters are in a fixed format, and the input parameters, control parameters and output parameters are configurable.

[0059] Table 1. Component template structure

[0060]

[0061] like Figure 2 As shown, the component template information configured by the user for each task in the visual interface is received, that is, the user's filling in of the component template structure in the visual interface is received to obtain the component template information, the component template information is parsed according to the template specification to obtain the corresponding component parameters, and the component is rendered in the pipeline orchestration interface according to the component parameter information to obtain the corresponding component, so that the user can intuitively see the generated component, and at the same time, a configuration file of the component is generated, and the configuration file is in YAML format or JSON format.

[0062] At this time, before receiving the component template information configured by the user for each task in the visual interface in step S1, the following steps are included:

[0063] S11, receiving the component type selected by the user for each task in the visual interface, determining whether the component type exists in the original component library, and if not, sending the component template information to be created to the user, wherein the component template information to be created is used by the user to create a new component;

[0064] S12. Receive the component template information to be created configured by the user for creating a new component, and generate configuration information to be confirmed in a preview form. When the configuration information to be confirmed is correct, send the configuration information to be confirmed to the original component library to realize the creation of the new component.

[0065] In this embodiment, if Figure 2 As shown, the component type selection made by the user for each task in the visual interface is received. If the component type selected by the user is not in the original component library, the component template information to be created is sent to the user, that is, the component template structure is sent to the user, and the component template information to be created configured by the user for creating a new component is received, that is, the filling of the template information to be created, that is, the component template structure, by the user is received, and the configuration information to be confirmed is generated in the form of a preview for the template information to be created sent by the user. When the user confirms that the configuration information to be confirmed is correct, the configuration information to be confirmed is sent to the original component library to realize the creation of a new component corresponding to the template information to be created filled in by the user.

[0066] Among them, the component types include: data preprocessing component, model training component, model evaluation component, model deployment component, data visualization component and notification component. Data preprocessing component: used to clean, deduplicate, convert format, enhance data and other operations on raw data to provide high-quality data input for downstream model training and evaluation; Model training component: used to load data, call training scripts, adjust hyperparameters, and output model files and log reports after completion, supporting common deep learning frameworks such as TensorFlow, PyTorch, Sklearn, etc.; Model evaluation component: used to load models and test data, evaluate the performance of models, and output indicator data: F1 score, recall rate, AUC, etc.; Model deployment component: used for online reasoning services, users can access the prediction results of the model through API; Data visualization component: used to visualize intermediate data into charts, reports and visual reports, so that users can intuitively understand the running status and intermediate results of the task; Notification component: used to send emails, WeChat or DingTalk messages when the pipeline is executed to a certain stage, so that users can know the running status of the pipeline.

[0067] At this time, the template specification in step S1 includes a first specification, a second specification and a third specification, and parsing the component template information according to the template specification includes:

[0068] S13, acquiring input parameters, output parameters, control parameters, basic task information and task parameters from the component template information;

[0069] S14. Parse the input parameter according to the first specification to obtain an input type of the input parameter, and obtain a component dynamic behavior according to the input type;

[0070] S15. Parse the output parameter according to the second specification to obtain an output type, and obtain a component output product according to the output type;

[0071] S16. Parse the control parameter according to the third specification to obtain parameter information of the control parameter, and obtain component execution behavior according to the parameter information;

[0072] S17: Use the component dynamic behavior, the component output product, the component execution behavior, the task basic information and the task parameters as component parameter information.

[0073] In this embodiment, if Figure 2As shown, the template specification includes a first specification, a second specification and a third specification. The input parameters are parsed according to the first specification to obtain the input type, and the dynamic behavior of the component is obtained according to the input type. The input types include: string, multi-line string, number, single choice, reference, switch and drop-down list. The output parameters are parsed according to the second specification to obtain the output type. The output product of the component is obtained according to the output type. The output types include: file and string. The control parameters are parsed according to the third specification to obtain the parameter information of the control parameters. The component execution behavior is obtained according to the parameter information. The component dynamic behavior, component output product, component execution behavior, task basic information and task parameters are used as component parameter information.

[0074] S2. Receive the user's behavioral operation on the component in the pipeline orchestration interface, generate a pipeline according to the behavioral operation, and run the configuration file of the component in the pipeline through the task scheduler to implement the execution of the pipeline.

[0075] In this embodiment, if Figure 2 As shown, the user's behavioral operations on components in the pipeline orchestration interface are received, including dragging components and connecting components. The components are connected by configuring the dependencies between components through connecting lines, so as to generate a pipeline according to the behavioral operations, and the configuration files of the components in the pipeline are scheduled to run according to the execution order of the pipeline through the task scheduler to realize the execution of the pipeline.

[0076] Example 2, please refer to Figure 3 The present invention provides a templated pipeline component design system 1, comprising: a task configuration module 2, a component creation module 21 and a task scheduling module 3.

[0077] Among them, the task configuration module 2 is used to receive the component template information configured by the user for each task in the visual interface, parse the component template information according to the template specification, obtain the corresponding component parameter information, render the component in the pipeline arrangement interface according to the component parameter information, obtain the corresponding component, and generate the configuration file of the component at the same time;

[0078] The task scheduling module 3 is used to receive the user's behavioral operations on the components in the pipeline scheduling interface, generate a pipeline according to the behavioral operations, and run the configuration files of the components in the pipeline through the task scheduler to implement the execution of the pipeline.

[0079] Specifically, the task configuration module 2 includes:

[0080] The component creation module 21 is used to receive the component type selected by the user for each task in the visual interface, and determine whether the component type exists in the original component library. If not, the component template information to be created is sent to the user, and the component template information to be created is used by the user to create a new component;

[0081] Receive the component template information to be created configured by the user for creating a new component, and generate configuration information to be confirmed in a preview form. When the configuration information to be confirmed is correct, send the configuration information to be confirmed to the original component library to realize the creation of the new component.

[0082] Specifically, the task configuration module 2 is specifically:

[0083] The template specification includes a first specification, a second specification and a third specification, and parsing the component template information according to the template specification includes:

[0084] Acquire input parameters, output parameters, control parameters, basic task information and task parameters from the component template information;

[0085] Parsing the input parameters according to the first specification to obtain component dynamic behavior, parsing the output parameters according to the second specification to obtain component output products, and parsing the control parameters according to the third specification to obtain component execution behavior;

[0086] The component dynamic behavior, the component output product, the component execution behavior, the task basic information and the task parameter are used as component parameter information.

[0087] Specifically, the component types include: data preprocessing component, model training component, model evaluation component, model deployment component, data visualization component and notification component.

[0088] The behavior operation includes dragging components and connecting components.

[0089] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0090] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0092] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0093] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0094] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0095] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A templated pipeline component design method, characterized in that: include: Receive the component template information configured by the user for each task in the visual interface, parse the component template information according to the template specification, obtain the corresponding component parameter information, render the component in the pipeline orchestration interface according to the component parameter information, obtain the corresponding component, and generate the configuration file of the component at the same time; Receive the user's behavioral operation on the component in the pipeline orchestration interface, generate a pipeline according to the behavioral operation, and run the configuration file of the component in the pipeline through the task scheduler to implement the execution of the pipeline.

2. A templated pipeline component design method as claimed in claim 1, characterized in that: The receiving of the component template information configured by the user for each task in the visual interface includes: Receive the component type selected by the user for each task in the visual interface, determine whether the component type exists in the original component library, and if not, send the component template information to be created to the user, where the component template information to be created is used by the user to create a new component; Receive the component template information to be created configured by the user for creating a new component, and generate configuration information to be confirmed in a preview form. When the configuration information to be confirmed is correct, send the configuration information to be confirmed to the original component library to realize the creation of the new component.

3. A templated pipeline component design method as claimed in claim 1, characterized in that: The template specification includes a first specification, a second specification and a third specification, and parsing the component template information according to the template specification includes: Acquire input parameters, output parameters, control parameters, basic task information and task parameters from the component template information; Parsing the input parameter according to the first specification to obtain an input type of the input parameter, and obtaining a component dynamic behavior according to the input type; Parsing the output parameter according to the second specification to obtain an output type, and obtaining a component output product according to the output type; Parsing the control parameter according to the third specification to obtain parameter information of the control parameter, and obtaining component execution behavior according to the parameter information; The component dynamic behavior, the component output product, the component execution behavior, the task basic information and the task parameter are used as component parameter information.

4. A templated pipeline component design method as claimed in claim 2, characterized in that: The component types include: data preprocessing component, model training component, model evaluation component, model deployment component, data visualization component and notification component.

5. A templated pipeline component design method as claimed in claim 1, characterized in that: The behavior operation includes dragging components and connecting components.

6. A templated pipeline component design system, characterized in that: include: The task configuration module is used to receive the component template information configured by the user for each task in the visual interface, parse the component template information according to the template specification, obtain the corresponding component parameter information, render the component in the pipeline orchestration interface according to the component parameter information, obtain the corresponding component, and generate the configuration file of the component at the same time; The task scheduling module is used to receive the user's behavioral operations on the components in the pipeline scheduling interface, generate a pipeline according to the behavioral operations, and run the configuration files of the components in the pipeline through the task scheduler to implement the execution of the pipeline.

7. A templated pipeline component design system as claimed in claim 6, characterized in that: The task configuration module includes: The component creation module is used to receive the component type selected by the user for each task in the visual interface, and determine whether the component type exists in the original component library. If not, the component template information to be created is sent to the user, and the component template information to be created is used by the user to create a new component; Receive the component template information to be created configured by the user for creating a new component, and generate configuration information to be confirmed in a preview form. When the configuration information to be confirmed is correct, send the configuration information to be confirmed to the original component library to realize the creation of the new component.

8. A templated pipeline component design system as claimed in claim 6, characterized in that: The task configuration module is specifically: The template specification includes a first specification, a second specification and a third specification, and parsing the component template information according to the template specification includes: Acquire input parameters, output parameters, control parameters, basic task information and task parameters from the component template information; Parsing the input parameter according to the first specification to obtain an input type of the input parameter, and obtaining a component dynamic behavior according to the input type; Parsing the output parameter according to the second specification to obtain an output type, and obtaining a component output product according to the output type; Parsing the control parameter according to the third specification to obtain parameter information of the control parameter, and obtaining component execution behavior according to the parameter information; The component dynamic behavior, the component output product, the component execution behavior, the task basic information and the task parameter are used as component parameter information.

9. A templated pipeline component design system as claimed in claim 7, characterized in that: The component types include: data preprocessing component, model training component, model evaluation component, model deployment component, data visualization component and notification component.

10. A templated pipeline component design system as claimed in claim 6, characterized in that: The behavior operation includes dragging components and connecting components.

Images