A method for building a machine learning workflow
By defining and reusing machine learning workflow templates, using Kubernetes cluster management technology, the complexity and management problems of workflow construction in the existing technology are solved, efficient development and deployment are achieved, and model management and resource use are simplified.
Patent Information
- Application Number
- CN202410428034.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-04-10
AI Technical Summary
The existing machine learning workflow construction is based on Spark technology, with unfriendly node support and inability to reuse workflow templates, resulting in difficulty in repetitive labor and model deployment management, and lack of version management and containerized cluster management.
By defining data loading, clustering, and classification model operators, using the @component() decorator to build operators, define the connection and dependencies between operators, compile and execute workflows, and use Kubernetes cluster management technology to realize workflows Version multiplexing and parameterization.
Improve development efficiency, simplify deployment and maintenance, support experiments and iterations, optimize resource management, promote team collaboration, and reduce duplicate labor and configuration errors.
Smart Images

Figure CN118378719B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method for constructing a machine learning workflow. Background Art
[0002] There are many technical solutions for constructing machine learning workflows, most of which are based on Spark. For example, the patent application with application number 201810487532.0 discloses a method for constructing a machine learning workflow based on Spark, including: pre-defining at least two workflow nodes, and setting a program code block corresponding to each of the workflow nodes; also including: displaying each of the workflow nodes to the user, and forming a directed flow graph including at least two target workflow nodes according to the user's triggering of each of the displayed workflow nodes; parsing the directed flow graph to determine the target program code blocks corresponding to each of the target workflow nodes from the set program code blocks, and determine the execution order between each of the target program code blocks; according to the execution order, selecting one of the unselected target program code blocks in turn, and sending the selected target program code block to the Spark cluster, so that the Spark cluster executes the target program code blocks it receives until each of the target program code blocks is received and successfully executed by the Spark.
[0003] The entire workflow is built based on Spark technology. The workflow does not support nodes of multiple computing frameworks. For example, the more well-known machine learning frameworks include TensorFlow, Caffe, Caffe2, PyTorch, MxNet, etc. It is unclear whether each node can run independently and whether the workflow needs to be rebuilt during operation. The parameter modification of the nodes included in the workflow cannot be recorded. There is no version management for the workflow used to track training after the model is deployed, which is not conducive to the life cycle management of the model. It is impossible to use containerized cluster management technology to implement running task scheduling. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a machine learning workflow construction method that allows modelers to quickly build new workflows and reduce repetitive work by reusing existing workflow templates.
[0005] The objective of the present invention is achieved through the following technical solution: A method for constructing a machine learning workflow, comprising the following steps:
[0006] Step 1: Construct an operator, including the following sub-steps:
[0007] Step 11. Define the data loading operator: The data loading operator uses @component() as a decorator and defines a function named load_data. @component() is placed above the defined function and can define the function's calculation logic, interface, and metadata.
[0008] Step 12. Define the kmeans clustering model operator: The function of the kmeans operator is to cluster the electric submersible pump working data loaded by the data loading operator into three categories: normal, faulty, and failed using the kmeans clustering model, and output the clustering results to MinIO storage; there are two different ways to output variables to MinIO storage in the operator function: one is to output as Output type, written as: 'output variable name: Output[output data type]'; the other is to output as OutputPath type, written as: 'output variable name: OutputPath[output data type]';
[0009] Step 13: Definition of data labeling and category balancing processing operators: Use the oversample operator to take the Kmeans model clustering results as category labels, count the clustering results, and balance the data of each category by upsampling; at the same time, divide the data into training sets and test sets according to the preset ratio, and save them in Minio;
[0010] Step 14: Construct a decision tree classification model operator: Use the decisionTree operator to take the clustering results after data balancing as the category label, construct a decisionTree classification model, and train it using the training set data; store the trained model and record the model metadata, including: model size, name, and calculation framework;
[0011] Step 15, definition of model prediction operator: use the predict operator, load the model file and test set saved in the previous step, use the model to predict and test the generalization performance of the test data, calculate the accuracy score, and output the confusion matrix; the confusion matrix 'contrusion_mat' is output as artifacts, and the format is "confusion_mat.log_confusion_matrix(category list: confusion matrix list)"; the elements 0, 2, and 1 in the confusion matrix correspond to the states "Normal", "Fault", and "Shut_down" of the electric submersible pump, respectively, indicating the three states of normal, fault, and failure;
[0012] Step 2: Define the connections and dependencies between operators. Define a directed acyclic graph (DAG), where nodes represent operators and edges represent dependencies between operators, which is achieved by specifying the output of one operator as the input of another operator.
[0013] Step 3: Compile the workflow to ensure that the workflow can be parsed and executed correctly, and save the defined workflow as a file;
[0014] Step 4: Execute the workflow and train the model.
[0015] Step 5. Submit the computing task and run the workflow as needed. Each time you run it, you need to fill in the workflow parameters. After filling in the relevant information, submit the defined workflow to the Kubernetes cluster and use the kubectl command line tool or the API provided by the workflow engine to create the corresponding Kubernetes resource objects.
[0016] In step 11, the data loading operator has the following characteristics:
[0017] (1) The decorator parameters include three: base_image, packages_to_install, and pip_index_urls. base_image refers to the runtime environment of this operator. packages_to_install is used to represent the list of packages that users need to install separately in addition to the packages already included in the base image. The type is list. pip_index_urls is used to represent the download address of the mirror source. The type is list.
[0018] (2) The load_data function defines an input datapath, which is the path for reading and storing data and is of str type. The datapath is set as a workflow parameter, and subsequent operators also use this parameter as input.
[0019] (3) The load_data function defines an output datafile, which is the Minio path for storing data and is of the InputPath type defined by dap, in order to pass data to downstream operators;
[0020] (4) The input / output parameters of the load_data function operator are assigned values only when the workflow is submitted for execution. Only the parameter name and type are determined during definition.
[0021] (5) Users can add a list of packages to be installed as needed, provided that the system's network connection allows it;
[0022] (6) The input of the load_data function must first define its data type, and its format is: 'input parameter name: data type';
[0023] (7) The main part of the load_data function is coded as usual.
[0024] In step 2, the pipeline function is used to define the connection and dependency between operators. The @dap.dsl.pipeline decorator function is placed on the pipeline function to define the pipeline. It contains two parameters, name and description, which define the name and description of the pipeline respectively.
[0025] The pipeline function is specifically the def esp_pipeline() function, which defines multiple workflow parameters in the function declaration: Workflow parameters refer to the input and output parameters specified in the workflow definition, which are used to describe the data transfer and dependency relationship between operators in the workflow; the input parameters of the workflow cannot contain Chinese characters;
[0026] After defining the connections and dependencies between operators, apply for pvc resources for the operators. When applying for pvc resources, you need to fill in the name of the pv, which is one of the storage volumes mounted on the current notebook. The initial name of the pvc is 'modify-sample'. Users can create their own folder names and modify the pvc name according to the read me.
[0027] When the operator mounts the pvc, the add_pvolumes method needs to be used. The parameter {' / mnt':vol} means that the requested vol is mounted in the / mnt directory in the container. Vol refers to the resource size of the pvc.
[0028] The specific implementation method of step 3 is: select dap.compiler.Compiler().compile() to compile the pipeline function and output the .yaml file; dap.compiler.Compiler().compile() needs to provide two parameters, the first is the previously defined pipeline function name, and the second is the output path of the .yaml file.
[0029] The beneficial effects of the present invention are:
[0030] 1. Improve development efficiency:
[0031] (1) By reusing existing workflow templates, modelers can quickly build new workflows and reduce duplication of work;
[0032] (2) Modelers can focus more on the implementation of business logic and algorithms instead of spending a lot of time on infrastructure and configuration.
[0033] 2. Simplify deployment and maintenance:
[0034] (1) The reused workflow version has been tested and verified, and has high stability and reliability; it can more accurately identify the working status of the ESP;
[0035] (2) This simplifies the deployment and maintenance process and reduces problems caused by configuration errors or version inconsistencies.
[0036] 3. Support experiments and iterations:
[0037] (1) In machine learning tasks, experiments and iterations are often required. By reusing workflow versions, data analysts can easily create multiple experimental versions, compare their results, and select the best one;
[0038] (2) It helps algorithm engineers quickly find the best model and parameter configuration.
[0039] 4. Resource management optimization: Reusing workflow versions helps optimize resource usage in Kubernetes clusters. By reusing defined Pod templates and configurations, unnecessary resource allocation and waste can be reduced.
[0040] 5. Promote teamwork:
[0041] (1) Workflow version reuse allows team members to share and use the same workflow templates and components, promoting team collaboration and knowledge sharing;
[0042] (2) Through unified workflow definitions, team members can better understand and collaborate to complete complex machine learning tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A flowchart of a method for constructing a machine learning workflow of the present invention;
[0044] Figure 2 A program defined for the data loading operator of the present invention;
[0045] Figure 3 A procedure defined for the kmeans clustering model operator of the present invention;
[0046] Figure 4 A definition program for the data labeling and category balancing processing operator of the present invention;
[0047] Figure 5 A definition program for the decision tree classification model operator of the present invention;
[0048] Figure 6 A definition procedure for the model prediction operator of the present invention;
[0049] Figure 7 A procedure for defining connections and dependencies between operators of the present invention;
[0050] Figure 8 A program for compiling the workflow of the present invention;
[0051] Fig. 9 A schematic diagram of the system interface for executing workflow;
[0052] Fig.10 A program for comparing and viewing workflow versions. DETAILED DESCRIPTION
[0053] The abbreviations and key terms of the present invention are defined as follows:
[0054] Machine learning workflow: A complete machine learning process generally includes four stages: data reading, ETL, feature engineering, model training, and evaluation. These four stages constitute an automated standard workflow.
[0055] Workflow parameters: When building a workflow, set the hyperparameters of some operators as workflow parameters. Each time you submit the workflow for execution, you can modify the hyperparameters and reuse the previous workflow to improve modeling efficiency.
[0056] Workflow version: Since the training model is based on the workflow, after the optimal model is applied to the production line, if the model needs to be iteratively optimized, it is necessary to track and iterate the workflow used to train the model at that time to obtain the latest model. The workflow tracking iteration process can generate multiple versions to facilitate model traceability.
[0057] Operator: The data processing tasks at each stage are completed by operators, which are the logical definitions of data calculations. Operators are connected through input and output, and ultimately form a complete workflow. The training model is based on the workflow and the work is submitted multiple times for training.
[0058] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.
[0059] A machine learning workflow construction method of the present invention uses containerization technology to build a workflow and define the computing framework of each operator. When defining each operator, the computing framework and version of each operator of base_image are required. DAP is a model development tool (python sdk) for the modeling platform, which is mainly used for operator development, workflow creation and submission and operation. The workflow library dap mainly includes: dap.Client class, dap.compiler, dap.component, dap.container, dap.dsl. The following takes the workflow of building an electric submersible pump fault diagnosis model as an example. This workflow mainly includes 5 operators. For the development of each operator, it is necessary to prepare: a Python script containing the execution logic; an interface that defines the operator; other metadata, including runtime environment, runtime commands and parameters, etc. Figure 1 As shown in the figure, the workflow construction specifically includes the following steps:
[0060] Step 1: Construct an operator, including the following sub-steps:
[0061] Step 11: Define the data loading operator: The data loading operator uses @component() as a decorator and defines a function named load_data. @component() is placed above the defined function and can define the function's calculation logic, interface, and metadata. The program for defining the data loading operator is as follows: Figure 2 shown.
[0062] The data loading operator has the following characteristics:
[0063] (1) The decorator parameters include three: base_image, packages_to_install and pip_index_urls; base_image refers to the runtime environment of this operator (the image of Pod, including all packages for the operator); the base_image of the present invention installs sklearn 0.20.3. Packages_to_install is used to represent the list of packages that users need to add separately in addition to the packages already included in the base image, and the type is list; pip_index_urls is used to represent the download address of the mirror source, and the type is list. It is recommended to use a domestic mirror source.
[0064] (2) The load_data function defines an input datapath, which is the path for reading and storing data and is of str type. The datapath is set as a workflow parameter, and subsequent operators also use this parameter as input.
[0065] (3) The load_data function defines an output datafile, which is the Minio path for storing data and is of the InputPath type defined by dap, in order to pass data to downstream operators;
[0066] (4) The input / output parameters of the load_data function operator are assigned values only when the workflow is submitted for execution. Only the parameter name and type are determined during definition.
[0067] (5) Users can add a list of packages to be installed as needed, provided that the system's network connection allows it;
[0068] (6) The input of the load_data function must first define its data type, and its format is: 'input parameter name: data type';
[0069] (7) The main part of the load_data function is coded as usual.
[0070] Step 12. Define the kmeans clustering model operator: The function of the kmeans operator is to use the kmeans clustering model to cluster the submersible pump working data loaded by the data loading operator into three categories: normal, faulty, and failed, and output the clustering results to MinIO storage; there are two different ways to output variables to MinIO storage in the operator function: one is to output as Output type, written as: 'output variable name: Output[output data type]'; the other is to output as OutputPath type, written as: 'output variable name: OutputPath[output data type]'; both store the output to MinIO, the main difference lies in the amount of stored data. It is recommended to use the Output method for a small amount of data, and the output will be saved in json format. It is recommended to use the OutputPath method for a large amount of data output. The program defined by the kmeans clustering model operator is as follows Figure 3 shown.
[0071] Step 13, data labeling and definition of category balance processing operators: use the oversample operator to take the Kmeans model clustering results as category labels, and count the clustering results. If there is a category imbalance, balance the data of each category by upsampling; at the same time, divide the data into training set and test set according to the preset ratio, and save them in Minio; the definition procedure of data labeling and category balance processing operators is as follows: Figure 4 shown.
[0072] Step 14: Construct a decision tree classification model operator: Use the decisionTree operator to take the balanced clustering results as the category label, construct a decisionTree classification model, and train it using the training set data; store the trained model and record the model metadata, including: model size, name, and calculation framework; the definition procedure of the decision tree classification model operator is as follows: Figure 5 shown.
[0073] illustrate:
[0074] The model is saved by serializing the trained optimal model through joblib, and the name of the saved file is model.joblib (for subsequent deployment, the model file name under the sklearn framework must be the model name).
[0075] Model output artifacts output_model.name, output_model.metadata{size,modelpath}.
[0076] Step 15, definition of model prediction operator: use the predict operator, load the model file and test set saved in the previous step, use the model to predict and test the generalization performance of the test data, calculate the accuracy score, and output the confusion matrix; the confusion matrix 'contrusion_mat' is output as artifacts, and the format is "confusion_mat.log_confusion_matrix (category list: confusion matrix list)"; the elements 0, 2, and 1 in the confusion matrix correspond to the states "Normal", "Fault", and "Shut_down" of the electric submersible pump, respectively, indicating the three states of normal, fault, and failure; the definition procedure of the model prediction operator is as follows Figure 6 The confusion matrix and accuracy can be visualized in the visualization of the run details.
[0077] Step 2: Define the connections and dependencies between operators. Define a directed acyclic graph (DAG), where nodes represent operators and edges represent dependencies between operators. This is achieved by specifying the output of one operator as the input of another operator. For example, the kmeans_op operator is the output of the datafile operator and the input of the datalabel operator. In the subsequent execution process, when an operator is executed, its output parameters can be passed to subsequent operators as input parameters. The definition program is as follows: Figure 7 shown.
[0078] In step 2, the pipeline function is used to define the connection and dependency between operators. The @dap.dsl.pipeline decorator function is placed on the pipeline function to define the pipeline. It contains two parameters, name and description, which define the name and description of the pipeline respectively.
[0079] The pipeline function is specifically the def esp_pipeline() function, which defines multiple workflow parameters, such as datapath, test_size, etc.: Workflow parameters refer to the input and output parameters specified in the workflow definition, which are used to describe the data transfer and dependency between operators in the workflow; workflow parameters are parameters that are actually assigned at runtime. In the definition of the function declaration, the type of the parameter must be defined first, such as test_size:float, and then a virtual assignment is filled in, such as "=0.3", but the value is allowed to be changed by the user before running.
[0080] The input parameters of the workflow cannot contain Chinese characters. For example, the path type values such as modelpath cannot contain Chinese characters.
[0081] After defining the connections and dependencies between operators, apply for pvc resources for the operators. When applying for pvc resources, you need to fill in the name of the pv, which is one of the storage volumes mounted on the current notebook. The initial name of the pvc is 'modify-sample'. Users can create their own folder names and modify the pvc name according to the read me.
[0082] When the operator mounts the pvc, the add_pvolumes method needs to be used. The parameter {' / mnt':vol} means that the requested vol is mounted in the / mnt directory in the container. Vol refers to the resource size of the pvc.
[0083] Workflow parameters main advantages:
[0084] 1) Flexibility: By defining workflow parameters, users can dynamically adjust the workflow input data, configuration options, and output requirements according to different needs and scenarios. This makes the workflow more flexible and able to adapt to the needs of various modeling tasks.
[0085] 2) Reusability: By parameterizing workflows, users can create common workflow templates and generate different versions, and reuse them in different tasks. Simply adjust the corresponding parameter values to adapt to different data sets, models, or algorithms. This improves the reusability of workflows and reduces duplication of work.
[0086] 3) Maintainability: Clearly defined workflow parameters help improve code readability and maintainability. When other developers view or modify the workflow, they can clearly understand the role and purpose of each parameter, making it easier to debug and optimize the code.
[0087] 4) Monitoring and debugging: By parameterizing the workflow, users can more easily monitor and debug the execution process of the workflow. The input and output values of each parameter can be recorded to facilitate troubleshooting and locating problems when they occur. In addition, different log levels and output formats can be set through parameterization to better understand the execution of the workflow.
[0088] 5) Performance optimization: Users can optimize the performance of workflows by adjusting workflow parameters. For example, they can adjust parameters such as parallelism and resource allocation according to the needs and limitations of computing resources to improve the execution efficiency and throughput of workflows.
[0089] Step 3: Compile the workflow to ensure that the workflow can be parsed and executed correctly, and save the defined workflow as a file; Figure 8 As shown. The specific implementation method is: select dap.compiler.Compiler().compile() to compile the pipeline function and output the .yaml file; dap.compiler.Compiler().compile() needs to provide two parameters, the first is the name of the pipeline function defined previously, and the second is the output path of the .yaml file. It is worth noting that the mode needs to be specified in Compile to specify the version of the pipeline to be used. The default version is V1. Here, V2 is selected, mode = dap.dsl.PipelineExecutionMode.V2_COMPATIBLE;
[0090] Step 4: Execute the workflow and train the model. This process can be operated in the system interface. Create a new workflow and select the compiled YAML file, such as Fig. 9 shown.
[0091] Successfully created, visualized workflow. Workflow supports different version management. For workflow version management, each time a workflow is generated, a version needs to be defined. Different versions can be compared and viewed. Fig.10 shown.
[0092] Through workflow version management, users can utilize previously defined and tested workflows to avoid duplication of work, improve efficiency, and promote team collaboration.
[0093] (1) The system supports defining common or complex workflows as templates and storing them in the system. When a similar workflow needs to be executed, a new instance can be created directly from the template and parameters can be adjusted as needed.
[0094] (2) By using parameters in workflow definitions, workflows are made more flexible and reusable. Different running instances can determine the behavior of the workflow by providing different parameter values.
[0095] (3) Each version of the workflow can be marked, tracked, and rolled back, ensuring that workflow changes are controllable and traceable.
[0096] Step 5. Submit the computing task and run the workflow as needed. Each time you run it, you need to fill in the workflow parameters. After filling in the relevant information, submit the defined workflow to the Kubernetes cluster and use the kubectl command line tool or the API provided by the workflow engine to create the corresponding Kubernetes resource objects.
[0097] Each operator in the workflow defines the required operating environment, dependencies, code, etc. When the user submits a workflow execution request in the front-end interface, the system will create a corresponding Pod based on the operator information in the workflow definition. Pod is the smallest deployment unit in Kubernetes and contains one or more closely related containers. Each operator corresponds to a Pod, and all executed Pods in the system are scheduled and managed by Kubernetes.
[0098] After the workflow is established, you can also perform the following operations:
[0099] Step 6: After the workflow is executed, you can view the input / output results and operation log of each operator.
[0100] Step 7. Select a suitable model for storage based on the running results. After training is completed, collect the evaluation results of the model, such as accuracy, loss value and other indicators, and export the trained model to a deployable ONNX format according to the specifications of the training framework. Serialization is performed during the generation of the model file to reduce the model size and facilitate storage and transmission. The system uses MinIO to store serialized model files. Model files implement version control to ensure that each release has a clear version tag for easy backtracking and management.
[0101] Step 8: Deploy the stored model and configure the model service to expose the RESTful API so that external clients can perform model inference through HTTP requests.
[0102] The present invention builds workflows based on Kubernetes containerized cluster management technology. The workflows support version management and realize workflow version reuse.
[0103] The operators that make up the machine learning workflow can all be run as separate containers during operation. In the container, you can define its computing framework separately and encapsulate the operation dependencies of this operator into an image. After the training is completed, you can view its operation log and status in each container.
[0104] Operator parameters can be set as workflow parameters. One of the most difficult parts of the machine learning workflow is finding optimal hyperparameters for the model. The performance of the model is directly related to the hyperparameters. The better the hyperparameters are tuned, the better the model effect will be. Therefore, without modifying the workflow, you only need to modify the workflow parameters set for each training process.
[0105] The workflow supports different version management. For multiple model training processes, how to automate some modeling content, just modify the corresponding parameters to submit the model training, and how to trace the source of the model if it needs to be updated and iterated after the model is applied to the production line. When tracing back to all the workflow versions of the model at that time, optimizing and iterating on the source version will be more efficient and the model quality will be better.
[0106] When the workflow is submitted for execution, you can choose to run it on a scheduled basis, automatically triggering the model to run according to certain conditions, reducing manual involvement and improving work efficiency.
[0107] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.
Claims
1. A method for constructing a machine learning workflow, characterized in that: The following steps are involved: Step 1: Construct an operator, including the following sub-steps: Step 11. Define the data loading operator: The data loading operator uses @component() as a decorator and defines a function named load_data. @component() is placed above the defined function and can define the function's calculation logic, interface, and metadata. Step 12. Define the kmeans clustering model operator: The function of the kmeans operator is to cluster the electric submersible pump working data loaded by the data loading operator into three categories: normal, faulty, and failed using the kmeans clustering model, and output the clustering results to MinIO storage; there are two different ways to output variables to MinIO storage in the operator function: one is to output as Output type, written as: 'output variable name: Output[output data type]'; the other is to output as OutputPath type, written as: 'output variable name: OutputPath[output data type]'; Step 13: Definition of data labeling and category balancing processing operators: Use the oversample operator to take the Kmeans model clustering results as category labels, count the clustering results, and balance the data of each category by upsampling; at the same time, divide the data into training sets and test sets according to the preset ratio, and save them in Minio; Step 14: Construct a decision tree classification model operator: Use the decisionTree operator to take the balanced clustering result of the data as the category label, construct a decisionTree classification model, and use the training set data for training; Store the trained model and record the model metadata, including: model size, name, and computing framework; Step 15, definition of model prediction operator: use the predict operator, load the model file and test set saved in the previous step, use the model to test the prediction and generalization performance of the test data, calculate the accuracy score, and output the confusion matrix; the confusion matrix 'contrusion_mat' is output as artifacts, and the format is "confusion_mat.log_confusion_matrix(category list: confusion matrix list)"; the elements 0, 2, and 1 in the confusion matrix correspond to the states "Normal", "Fault", and "Shut_down" of the electric submersible pump, respectively, indicating the three states of normal, fault, and failure; Step 2: Define the connections and dependencies between operators. Define a directed acyclic graph (DAG), where nodes represent operators and edges represent dependencies between operators, which is achieved by specifying the output of one operator as the input of another operator. Step 3: Compile the workflow to ensure that the workflow can be parsed and executed correctly, and save the defined workflow as a file; Step 4: Execute the workflow and train the model. Step 5. Submit the computing task and run the workflow as needed. Each time you run it, you need to fill in the workflow parameters. After filling in the relevant information, submit the defined workflow to the Kubernetes cluster and use the kubectl command line tool or the API provided by the workflow engine to create the corresponding Kubernetes resource objects.
2. A method for constructing a machine learning workflow according to claim 1, characterized in that: In step 11, the data loading operator has the following characteristics: (1) The decorator parameters include three: base_image, packages_to_install, and pip_index_urls. base_image refers to the runtime environment of this operator. packages_to_install is used to represent the list of packages that users need to install separately in addition to the packages already included in the base image. The type is list. pip_index_urls is used to represent the download address of the mirror source. The type is list. (2) The load_data function defines an input datapath, which is the path for reading and storing data and is of str type. The datapath is set as a workflow parameter, and subsequent operators also use this parameter as input. (3) The load_data function defines an output datafile, which is the Minio path for storing data and is of the InputPath type defined by dap, in order to pass data to downstream operators; (4) The input / output parameters of the load_data function operator are assigned values only when the workflow is submitted for execution. Only the parameter name and type are determined during definition. (5) Users can add a list of packages to be installed as needed, provided that the system's network connection allows it; (6) The input of the load_data function must first define its data type, and its format is: 'input parameter name: data type'; (7) For the main part of the load_data function, write the code as usual.
3. A method for constructing a machine learning workflow according to claim 1, characterized in that: In step 2, the pipeline function is used to define the connection and dependency between operators. The @dap.dsl.pipeline decorator function is placed on the pipeline function to define the pipeline. It contains two parameters, name and description, which define the name and description of the pipeline respectively. The pipeline function is specifically the def esp_pipeline() function, which defines multiple workflow parameters in the function declaration: Workflow parameters refer to the input and output parameters specified in the workflow definition, which are used to describe the data transfer and dependency relationship between operators in the workflow; the input parameters of the workflow cannot contain Chinese characters; After defining the connections and dependencies between operators, apply for pvc resources for the operators. When applying for pvc resources, you need to fill in the name of the pv, which is one of the storage volumes mounted on the current notebook. The initial name of the pvc is 'modify-sample'. Users can create their own folder names and modify the pvc name according to the read me. When the operator mounts the pvc, the add_pvolumes method needs to be used. The parameter {' / mnt':vol} means that the requested vol is mounted in the / mnt directory in the container. Vol refers to the resource size of the pvc.
4. A method for constructing a machine learning workflow according to claim 3, characterized in that: The specific implementation method of step 3 is: select dap.compiler.Compiler().compile() to compile the pipeline function and output a .yaml file; dap.compiler.Compiler().compile() requires two parameters, the first is the name of the pipeline function defined previously, and the second is the output path of the .yaml file.
Citation Information
Patent Citations
Machine learning workflow construction method and device based on Spark
CN108733358A
Machine learning service assembly method based on Mahout
CN107169572A
Machine learning model automatic production line construction method and system
CN114115857A
Visual full-process machine learning platform, control method, client and application
CN114139728A
Construction method of lightweight Spark pipeline prediction model
CN115809069A
