Deployable pipeline generation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a deployable pipeline generation method, device and equipment and a storage medium, the method comprises the following steps: using a directed acyclic graph modeling tool to create a plurality of workflow nodes, and configuring the plurality of workflow nodes to obtain a workflow, the plurality of workflow nodes comprise operation types, data types and input and output parameters, applying a model evaluation analysis strategy and a model construction algorithm to the training data set to obtain a plurality of model results, selecting a deployment model by comparing the plurality of model results, obtaining a dependency relationship of the deployment model from the workflow, and establishing a deployment model; and generating a deployable pipeline according to the dependency relationship. The workflow is created by using the directed acyclic graph modeling tool, the deployment model is selected by comparing the multiple model results, and the deployable pipeline is generated according to the dependency relationship, so that the accuracy and the high efficiency of the deployable pipeline generation process are ensured, and the complexity and the error rate of the deployable pipeline generation process are reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a deployable pipeline generation method, device, equipment and storage medium. Background Art

[0002] Workflow technology models, executes, manages and monitors work processes in a computerized manner, usually involving organizing a series of tasks, activities or decisions in a specific logical order and rules to ensure that the work is carried out according to predetermined standards and processes. Pipeline technology refers to connecting multiple data processing or computing tasks in a certain order and dependency relationship to form a continuous data processing process.

[0003] Nowadays, the technology of using directed acyclic graph (DAG) to model workflows and pipeline generation has been widely used, but there are still defects and shortcomings in actual use. For example, users need to manually adjust the dependency paths (i.e., the sequential relationship between workflow nodes) between multiple workflow nodes (such as data input nodes, preprocessing nodes, etc.), which is prone to configuration errors and dependency confusion. The existing workflow deployment method lacks dynamic response capabilities and cannot automatically generate new pipelines based on changes. In addition, the system migration is more complex and may encounter compatibility or scalability issues. Summary of the invention

[0004] The main purpose of this application is to provide a deployable pipeline generation method, device, equipment and storage medium, aiming to solve the technical problems that manual configuration and reliance on workflow tools can easily lead to configuration errors and dependency confusion, and cannot dynamically respond to data and model changes.

[0005] To achieve the above objectives, the present application proposes a deployable pipeline generation method, the method comprising:

[0006] Using a directed acyclic graph modeling tool to create a plurality of workflow nodes, and obtaining a workflow by configuring the plurality of workflow nodes, the plurality of workflow nodes including operation types, data types, and input and output parameters;

[0007] Applying the model evaluation analysis strategy and the model building algorithm to the training data set to obtain multiple model results, and selecting a deployment model by comparing the multiple model results;

[0008] Dependencies of the deployment model are obtained from the workflow, and a deployable pipeline is generated according to the dependencies.

[0009] In one embodiment, the step of obtaining the dependency of the deployment model from the workflow and generating a deployable pipeline according to the dependency includes:

[0010] According to the topological sorting of the directed acyclic graph, traversing from the root node of the workflow, and obtaining the traversal result by recording the list of predecessor nodes and the operation type of each node;

[0011] Generate a dependency tree structure of the workflow according to the traversal result;

[0012] Identifying the dependency relationship of the deployment model from the dependency tree structure, and performing integrity verification on the dependency relationship to obtain a verification result;

[0013] The dependency is updated according to the verification result and the dependency optimization information, and a deployable pipeline is generated according to the dependency.

[0014] In one embodiment, the step of updating the dependency relationship according to the verification result and the dependency optimization information, and generating a deployable pipeline according to the dependency relationship includes:

[0015] Update the dependency relationship according to the verification result and the dependency optimization information;

[0016] Traversing the deployment model according to the dependency relationship to generate a node sequence of the deployment model;

[0017] Obtain each node processing unit by recording each node information of the deployment model, and encapsulate each node processing unit into each node processing module;

[0018] According to the order of the node sequence, the node processing modules are assembled into an initial deployable pipeline to generate a deployable pipeline.

[0019] In one embodiment, after the step of obtaining the dependency of the deployment model from the workflow and generating a deployable pipeline according to the dependency, the step further includes:

[0020] When receiving a model deployment request, build the operating environment on the deployment platform;

[0021] Based on the operating environment, the deployable pipeline is received, and according to the node sequence of the deployable pipeline, the configuration and parameters of each node processing module are parsed to obtain a parsing result;

[0022] According to the analysis result and the operation instruction script, execute each node of the deployable pipeline in sequence;

[0023] When executing the target model of the model deployment request, the output result is obtained by calling the target model.

[0024] In one embodiment, after the step of obtaining an output result by calling the target model when executing the target model of the model deployment request, the method further includes:

[0025] Comparing the initial output result with the expected result to obtain a comparison result;

[0026] When the comparison result is inconsistent, the target model is adjusted and called again to obtain a deployment result;

[0027] The deployment result is saved to a specified location, and a deployment log and a deployment execution report are generated;

[0028] The workflow is parsed and a new deployable pipeline is generated, and the new deployable pipeline is verified by comparing the deployable pipeline with the new deployable pipeline.

[0029] In one embodiment, the step of creating a plurality of workflow nodes using a directed acyclic graph modeling tool and obtaining a workflow by configuring the plurality of workflow nodes includes:

[0030] Creating a plurality of workflow nodes by using a directed acyclic graph modeling tool, and configuring parameters of the plurality of workflow nodes;

[0031] The dependency relationship of the plurality of workflow nodes is configured by a graphical method to form a directed acyclic graph structure, and the directed acyclic graph structure is used as the workflow.

[0032] In one embodiment, after the step of obtaining the dependency of the deployment model from the workflow and generating a deployable pipeline according to the dependency, the step further includes:

[0033] According to a preset output format, serializing the relevant information of the deployable pipeline into a character string in the preset output format;

[0034] The deployable pipeline is output, and a version number is assigned to the output deployable pipeline.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a deployable pipeline generation device, the deployable pipeline generation device comprising:

[0036] A workflow creation module, used to create multiple workflow nodes using a directed acyclic graph modeling tool, and obtain a workflow by configuring the multiple workflow nodes, wherein the multiple workflow nodes include operation types, data types, and input and output parameters;

[0037] A deployment model selection module is used to apply the model evaluation and analysis strategy and the model building algorithm to the training data set to obtain multiple model results, and select a deployment model by comparing the multiple model results;

[0038] The deployable pipeline generation module is used to obtain the dependency relationship of the deployment model from the workflow and generate a deployable pipeline according to the dependency relationship.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a deployable pipeline generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the deployable pipeline generation method as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the deployable pipeline generation method described above are implemented.

[0041] The technical solution proposed in this application uses a directed acyclic graph modeling tool to create multiple workflow nodes, and obtains a workflow by configuring multiple workflow nodes. The multiple workflow nodes include operation types, data types, and input and output parameters. The model evaluation and analysis strategy and the model building algorithm are applied to the training data set to obtain multiple model results, and the deployment model is selected by comparing the multiple model results. The dependency of the deployment model is obtained from the workflow, and a deployable pipeline is generated based on the dependency. Since this application creates a workflow by using a directed acyclic graph modeling tool, selects a deployment model by comparing multiple model results, and generates a deployable pipeline based on the dependency, the accuracy and efficiency of the deployable pipeline generation process are ensured, and the complexity and error rate of the deployable pipeline generation process are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A schematic diagram of a process flow provided for Embodiment 1 of the method for generating a deployable pipeline of the present application;

[0045] Figure 2 A flow chart of the second embodiment of the method for generating a deployable pipeline according to the present application;

[0046] Figure 3 Generate a flow diagram for the deployable pipeline for this application;

[0047] Figure 4 A flowchart diagram of Embodiment 3 of the method for generating a deployable pipeline according to the present application;

[0048] Figure 5 This is a schematic diagram of the module structure of a pipeline generation device that can be deployed in an embodiment of the present application;

[0049] Figure 6 Schematic diagram of the system structure of the hardware operating environment involved in the pipeline generation method that can be deployed in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] Existing deployable pipeline generation technologies are prone to configuration errors, dependency confusion and other problems, are highly complex, and cannot dynamically respond to data and model changes.

[0054] Therefore, in order to overcome the above-mentioned defects, the present application provides a solution by using a directed acyclic graph modeling tool to create a workflow, comparing multiple model results to select a deployment model, generating a deployable pipeline based on dependencies, ensuring the accuracy and efficiency of the deployable pipeline generation process, and reducing the complexity and error rate of the deployable pipeline generation process.

[0055] It should be noted that the execution subject of each embodiment of the present application may be a computing service system with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic system capable of realizing the above functions, a deployable pipeline generation system, etc. The following embodiments are described below taking the deployable pipeline generation system as an example.

[0056] Based on this, the present application embodiment provides a deployable pipeline generation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the deployable pipeline generation method of the present application.

[0057] In this embodiment, the deployable pipeline generation method includes steps S10 to S30:

[0058] Step S10: Use a directed acyclic graph modeling tool to create multiple workflow nodes, and obtain a workflow by configuring the multiple workflow nodes, wherein the multiple workflow nodes include operation types, data types, and input and output parameters.

[0059] It should be noted that a directed acyclic graph is a special graph structure in which the edges are directional and there is no path from a node back to the node through several edges (i.e. there is no cycle). This graph structure is often used to represent things with order or dependencies, such as workflows, task scheduling, etc. A directed acyclic graph modeling tool is a professional software application that provides a graphical user interface or programming interface, allowing users to easily create, modify and visualize directed acyclic graphs.

[0060] In addition, it should be noted that a workflow refers to a collection of interrelated tasks or activities that are performed in a specific order or rule to achieve a certain goal or complete a process. In workflow management, a node usually represents a specific task, operation, or decision point. In a directed acyclic graph, each node represents an operational step of data processing, feature engineering, or model training. Node types include but are not limited to data preprocessing (such as data cleaning, data filtering), feature selection (such as feature scaling, feature dimensionality reduction), model training (such as logistic regression, decision tree), model evaluation, etc. The edges between nodes represent the dependencies between tasks or the order of execution.

[0061] It should be understood that using modeling tools to add multiple nodes representing different tasks or activities in a directed acyclic graph, and to perform detailed settings and definitions on the multiple workflow nodes that have been created, including specifying the operation type, data type, and input and output parameters of the nodes, etc. These configurations will determine the specific functions of the workflow nodes and the execution order between them.

[0062] After completing the creation and configuration of the nodes, multiple workflow nodes are connected in a set order and dependency relationship through modeling tools to form a complete workflow. This process usually includes verifying the correctness of the workflow (such as checking whether there are circular dependencies, etc.) and generating a workflow definition that can be used for execution or management.

[0063] As can be seen from the above, step S10 may further include:

[0064] Creating a plurality of workflow nodes by using a directed acyclic graph modeling tool, and configuring parameters of the plurality of workflow nodes;

[0065] The dependency relationship of the plurality of workflow nodes is configured by a graphical method to form a directed acyclic graph structure, and the directed acyclic graph structure is used as the workflow.

[0066] It should be understood that parameter configuration can be to configure the input and output data types for multiple workflow nodes, such as the input is the original data table, the output is the cleaned data table, and the operation parameters of multiple workflow nodes can be set, such as the standardization method, principal component analysis parameters, etc. can be configured in the feature engineering node.

[0067] It should be noted that dependency refers to the association and dependence between nodes.

[0068] For example, assume that a software development project includes two parts, the front-end and the back-end. When deploying this project, a deployable pipeline is designed, including code submission nodes, build nodes, test nodes, deployment front-end nodes, deployment back-end nodes, integrated test nodes, etc. There are clear dependencies between the nodes, for example, the build node depends on the code submission node, the test node depends on the build node, etc., that is, the build task will be triggered only after the code is submitted; and the test will be performed only after the build is successful.

[0069] It is understandable that in the graphical interface, the order and dependency between nodes can be defined by dragging, connecting, and other operations, thereby forming a directed acyclic graph structure, and the formed directed acyclic graph structure is used as a workflow.

[0070] It should be understood that the dependencies between multiple workflow nodes are configured into the workflow so that the directed acyclic graph modeling tool can form a complete directed acyclic graph structure. Each dependency line represents the transfer path of data or operations, such as the output of the feature selection node as the input of the model training node.

[0071] Step S20, applying the model evaluation and analysis strategy and the model building algorithm to the training data set to obtain multiple model results, and selecting a deployment model by comparing the multiple model results.

[0072] It should be noted that the model evaluation and analysis strategy refers to a series of techniques and means for processing, interpreting and inferring data, such as descriptive statistics, correlation analysis, hypothesis testing, etc., which are used to extract useful information from data. The model building algorithm is a mathematical model or calculation rule, such as decision trees, random forests, deep learning, etc., which is used to process data. The training data set is a data set used to train machine learning models, usually containing a large number of input-output pairs, which are used to learn model parameters.

[0073] It should be understood that model results refer to the results obtained by applying model evaluation analysis strategies and model building algorithms to training data sets, usually in the form of predicted values, classification labels or clustering results, etc. Deployed models refer to models that have been trained, verified and selected for final use, which will be deployed in a specific environment for prediction or decision making on new data.

[0074] By applying model evaluation and analysis strategies and model building algorithms, multiple model results are obtained from the training data set. Multiple model results are compared and analyzed to evaluate their performance, such as accuracy, recall rate, etc., and the model with the best performance or that meets specific needs is selected as the deployment model.

[0075] Step S30, obtaining the dependency of the deployment model from the workflow, and generating a deployable pipeline according to the dependency.

[0076] It should be noted that a deployable pipeline is an automated process or pipeline used to deploy an application or system from a development environment to a production environment. It usually includes multiple stages such as building, testing, and deployment, and can automatically perform tasks in these stages.

[0077] It should be understood that extracting or querying the dependencies of the deployment model from the workflow usually involves parsing the definition or configuration of the workflow to identify the dependencies between various nodes or elements. Based on the extracted dependencies, a deployable pipeline is constructed.

[0078] In order to ensure the consistency and readability of the output deployable pipeline information and facilitate tracking and management of different versions of deployable pipelines, after step S30, the following may also be included:

[0079] According to a preset output format, serializing the relevant information of the deployable pipeline into a character string in the preset output format;

[0080] The deployable pipeline is output, and a version number is assigned to the output deployable pipeline.

[0081] It should be noted that the preset output format is usually a standardized data representation method, such as JSON format, YAML format, etc. Serialization is the process of converting data structure or object state into a format that can be stored or transmitted. Through serialization, complex deployable pipeline information can be converted into a string format that is easy to store, transmit and parse.

[0082] It should be understood that the relevant information of the deployable pipeline is converted into a string in a preset output format, and the serialized string is written to a file, a database, or sent to other systems over a network. In addition, in order to facilitate tracking the update history of the deployable pipeline when the model changes, a version number is assigned to the output deployable pipeline.

[0083] This embodiment uses a directed acyclic graph modeling tool to create multiple workflow nodes, and obtains a workflow by configuring multiple workflow nodes, wherein the multiple workflow nodes include operation types, data types, and input and output parameters, and applies the model evaluation and analysis strategy and the model building algorithm to the training data set to obtain multiple model results, and selects a deployment model by comparing multiple model results, obtains the dependency of the deployment model from the workflow, and generates a deployable pipeline based on the dependency. Since this application creates a workflow by using a directed acyclic graph modeling tool, selects a deployment model by comparing multiple model results, and generates a deployable pipeline based on the dependency, the accuracy and efficiency of the deployable pipeline generation process are ensured, and the complexity and error rate of the deployable pipeline generation process are reduced.

[0084] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , the step S30 may include steps S301 to S304:

[0085] Step S301 , traversing from the root node of the workflow according to the topological sorting of the directed acyclic graph, and obtaining the traversal result by recording the preceding node list and operation type of each node.

[0086] It should be noted that topological sorting is a method of sorting all nodes in a directed acyclic graph, so that for each directed edge (u, v) in the graph, node u appears before node v in the sort. In a directed graph, a node without a predecessor node (i.e., a node with no edge pointing to it) is called a root node. The predecessor node list refers to a list of all nodes that point to a node in the graph through directed edges. The operation type refers to the type of operation or task represented by the node, such as calculation, data transmission, decision-making, etc.

[0087] Visit each node in the graph in the order of topological sorting, save the information obtained during the traversal process (such as the list of predecessor nodes and operation types), and after the traversal is completed, organize all the recorded information into the required traversal results.

[0088] It should be noted that the information recorded during the traversal operation can be the input, output and dependent predecessor nodes of each node, and the input data set, operation type (such as data processing, feature selection, model training), output data set, etc. that each node depends on.

[0089] Step S302: Generate a dependency tree structure of the workflow according to the traversal result.

[0090] It should be noted that a dependency tree is a data structure used to represent the dependency relationship between elements (such as tasks, activities, etc.). In the tree structure, each node represents an element, and the lines (or edges) between nodes represent the dependency relationship between elements. If an element depends on another element, the dependent element will appear in the tree structure as a child node of the dependent element.

[0091] It should be understood that based on the node dependencies, a dependency tree structure of the workflow is generated, where the root node is the data preprocessing node and the leaf node is the model deployment node. In addition, the node dependencies need to be deduplicated and optimized to ensure that the path of each dependency is unique and loop-free.

[0092] Step S303: Identify the dependency relationship of the deployment model from the dependency tree structure, and perform integrity verification on the dependency relationship to obtain a verification result.

[0093] It can be understood that integrity verification refers to checking the dependencies of the deployed model to ensure that all dependencies are correct. That is, after the deployment model is determined, the dependencies of all its predecessor nodes are traced back and all intermediate nodes from the input data node to the model node are verified.

[0094] Step S304: update the dependency relationship according to the verification result and the dependency optimization information, and generate a deployable pipeline according to the dependency relationship.

[0095] It is understandable that the dependency optimization information may be an improvement suggestion or optimization solution obtained after analyzing the existing dependency, such as merging the same data processing nodes, reducing redundant operation steps, etc. The existing dependency is modified or optimized according to the verification results and the dependency optimization information, and a deployable pipeline is created according to the updated dependency.

[0096] Furthermore, in order for each node to cooperate with each other in an optimal manner and improve the accuracy of the deployable pipeline, the step S304 may include:

[0097] Update the dependency relationship according to the verification result and the dependency optimization information;

[0098] Traversing the deployment model according to the dependency relationship to generate a node sequence of the deployment model;

[0099] Obtain each node processing unit by recording each node information of the deployment model, and encapsulate each node processing unit into each node processing module;

[0100] According to the order of the node sequence, the node processing modules are assembled into an initial deployable pipeline to generate a deployable pipeline.

[0101] It should be noted that the node sequence refers to the set of nodes arranged according to a certain rule or order when traversing the deployment model. The node processing unit refers to the basic unit for processing each node in the deployment model, which may include data parsing, logic processing, resource allocation and other operations. The node processing module refers to the module that encapsulates the node processing unit, which usually has higher reusability and maintainability.

[0102] It can be understood that according to the verification results and dependency optimization information, the original dependency is modified or adjusted, each node in the deployment model is accessed according to a certain rule or sequence, and a node sequence is generated according to the traversal results.

[0103] It should be understood that the information of each node in the deployment model is recorded, such as the operation type, input data and output data of the node as each node processing unit. The node processing unit is encapsulated as a node processing module, and each node processing module is assembled into the initial deployable pipeline in the order of the node sequence.

[0104] For ease of understanding, refer to Figure 3 This is for illustration, but does not limit the present application. Assuming that the model node selected to be deployed is the LGBM classification model, the parsed dependent nodes are data loading nodes, missing value filling nodes, and feature derivation nodes. Extract and save the operation type, input and output parameters, operation method and other information of the node as a node processing unit, encapsulate each node processing unit into each node processing module, load each node processing module into the initial deployable pipeline, and generate a deployable pipeline.

[0105] This embodiment generates a traversal result of the workflow based on the topological sorting of the directed acyclic graph, builds a dependency tree structure of the workflow according to the traversal result, identifies and updates the dependency relationship of the deployment model from the structure, obtains the node processing unit by recording the node information of each node of the deployment model, and encapsulates it into a node processing module, and assembles the node processing modules into a deployable pipeline in the order of the node sequence, thereby ensuring the accuracy and reliability of the deployment pipeline generation and making the whole process more efficient and maintainable.

[0106] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 After step S30, steps S311 to S314 may also be included:

[0107] Step S311, upon receiving a model deployment request, building an operating environment on the deployment platform.

[0108] It is understandable that model deployment usually refers to running the trained model in a specific environment, which needs to meet all the conditions for model operation, including hardware resources, software framework, dependent libraries, etc. Therefore, building a running environment on the deployment platform is a necessary step to ensure that the model can run normally.

[0109] Step S312: Based on the operating environment, the deployable pipeline is received, and the configuration and parameters of each node processing module are parsed according to the node sequence of the deployable pipeline to obtain a parsing result.

[0110] It is understood that in the operating environment, the information of the deployable pipeline received from a source (such as a file, a database, an application program interface, etc.) may include the order of nodes, the type of each node, and the configuration and parameters of each node.

[0111] Process each node one by one in the order of the nodes, including reading its configuration and parameters. The configuration and parameters may exist in various forms, such as JSON, YAML, XML, etc., parse these configurations and parameters into a format that the program can understand, store the parsing results of each node, and directly use the parsing results for subsequent processing steps, such as executing a deployable pipeline.

[0112] Step S313: Execute each node of the deployable pipeline in sequence according to the parsing result and the operation instruction script.

[0113] It should be noted that the operation instruction script may be a script file (such as a Shell script, Python script, etc.) for executing a specific task, or an instruction to call an external tool. Calling the task or instruction defined in the operation instruction script to perform the operation of the current node may involve calling an external tool, executing a script file, or interacting with a remote service.

[0114] Step S314, when executing to the target model of the model deployment request, obtaining the output result by calling the target model.

[0115] It is understandable that when executing the target model of the model deployment request, the target model can be called through an application program interface or a model file.

[0116] In order to timely discover and correct deviations in model deployment, ensure the accuracy of deployment results, and facilitate subsequent tracking and auditing, after step S314, the following may also be included:

[0117] Comparing the initial output result with the expected result to obtain a comparison result;

[0118] When the comparison result is inconsistent, the target model is adjusted and called again to obtain a deployment result;

[0119] The deployment result is saved to a specified location, and a deployment log and a deployment execution report are generated;

[0120] The workflow is parsed and a new deployable pipeline is generated, and the new deployable pipeline is verified by comparing the deployable pipeline with the new deployable pipeline.

[0121] It should be noted that deployment logs are log files that record events, errors, and warnings that occur during the deployment process. Deployment execution reports are reports that provide an overview of the deployment process, results, and possible recommendations. Expected results can be pre-defined, based on historical data, or correct results obtained through other means.

[0122] The deployment results of the model (such as prediction results, output data tables) are saved to the specified database or file system, and a deployment log and deployment execution report are generated to record the execution status and execution time of each node in the deployable pipeline.

[0123] It should be understood that in order to check the accuracy of the model output and ensure that it meets expectations, the output results are compared with the expected results. If the comparison results show that the output results are inconsistent with the expected results, it means that the model may have errors or need to be optimized. The target model needs to be adjusted. The adjustment may involve modifying the model parameters, updating the training data, or improving the model algorithm, etc. After the adjustment is completed, the model is called again to obtain a new output result.

[0124] It is understood that the adjusted model (or the initial model, if the output is consistent with expectations) is deployed to the specified environment. Parse the existing workflow, which usually involves understanding the steps in the workflow, their dependencies, and the inputs and outputs required for each step, etc. Based on the parsing results, generate a new deployable pipeline. The new pipeline may contain an optimized model, new data processing steps, or an improved workflow.

[0125] Verify the correctness and effectiveness of the new deployable pipeline by comparing it with the existing deployable pipeline. The comparison can include checking the structure of the new deployable pipeline, the order of steps, the implementation of each step, etc. You can also run the new pipeline and collect its output results and performance indicators to further verify its performance.

[0126] It should be understood that if the verification results show that the new deployable pipeline meets expectations, it can be subsequently deployed and used. If there are problems, it is necessary to return to the previous steps for further adjustment and optimization.

[0127] It should be noted that the deployable pipeline generation method of the present application does not rely on a specific platform and can support a variety of deployment environments (such as local servers, cloud platforms, Kubernetes clusters, etc.). It supports the application of automatically generated deployable pipelines in different deployment environments (such as different cloud platforms or containerized environments) without reconfiguration and adaptation.

[0128] This embodiment builds a running environment on the deployment platform to respond to model deployment requests, receives and parses the node processing module configuration and parameters of the deployable pipeline, executes the pipeline nodes in sequence according to the parsing results and the operation instruction script, obtains the output result when the target model is executed, compares the initial output result with the expected result, and adjusts and calls the target model again if there is any inconsistency until a satisfactory deployment result is obtained, and verifies the new deployable pipeline, thereby improving the success rate and quality of model deployment.

[0129] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the deployable pipeline generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0130] This application also provides a deployable pipeline generation device, please refer to Figure 5 , the deployable pipeline generating device comprises:

[0131] The workflow creation module 10 is used to create a plurality of workflow nodes using a directed acyclic graph modeling tool, and obtain a workflow by configuring the plurality of workflow nodes, wherein the plurality of workflow nodes include operation types, data types, and input and output parameters;

[0132] A deployment model selection module 20 is used to apply the model evaluation and analysis strategy and the model building algorithm to the training data set to obtain multiple model results, and select a deployment model by comparing the multiple model results;

[0133] The deployable pipeline generation module 30 is used to obtain the dependency relationship of the deployment model from the workflow and generate a deployable pipeline according to the dependency relationship.

[0134] The deployable pipeline generation device provided by the present application adopts the deployable pipeline generation method in the above embodiment, which can solve the technical problems that the manual configuration and reliance on workflow tools easily lead to configuration errors and dependency confusion, and cannot dynamically respond to data and model changes. Compared with the prior art, the beneficial effects of the deployable pipeline generation device provided by the present application are the same as the beneficial effects of the deployable pipeline generation method provided by the above embodiment, and the other technical features in the deployable pipeline generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0135] The present application provides a deployable pipeline generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deployable pipeline generation method in the above-mentioned embodiment one.

[0136] Reference below Figure 6 , which shows a schematic diagram of the structure of a deployable pipeline generation device suitable for implementing the embodiments of the present application. The deployable pipeline generation device in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Desctions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The deployable pipeline generation device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0137] like Figure 6As shown, the deployable pipeline generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the deployable pipeline generation device are also stored in RAM1004. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following devices may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the deployable pipeline generation device to communicate wirelessly or wired with other systems to exchange data. Although the deployable pipeline generation device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0138] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0139] The deployable pipeline generation device provided by the present application adopts the deployable pipeline generation method in the above embodiment, which can solve the technical problems that manual configuration and reliance on workflow tools easily lead to configuration errors and dependency confusion, and cannot dynamically respond to data and model changes. Compared with the prior art, the beneficial effects of the deployable pipeline generation device provided by the present application are the same as the beneficial effects of the deployable pipeline generation method provided by the above embodiment, and the other technical features in the deployable pipeline generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0140] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0142] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the deployable pipeline generation method in the above-mentioned embodiment.

[0143] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0144] The computer-readable storage medium may be included in the deployable pipeline generation system; or may exist independently without being assembled into the deployable pipeline generation system.

[0145] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the deployable pipeline generation system, the deployable pipeline generation system: creates multiple workflow nodes using a directed acyclic graph modeling tool, and obtains a workflow by configuring the multiple workflow nodes, wherein the multiple workflow nodes include operation types, data types, and input and output parameters, applies a model evaluation and analysis strategy and a model building algorithm to a training data set, obtains multiple model results, selects a deployment model by comparing the multiple model results, obtains the dependency of the deployment model from the workflow, and generates a deployable pipeline based on the dependency.

[0146] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0149] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned deployable pipeline generation method, which can solve the technical problems that manual configuration and reliance on workflow tools easily lead to configuration errors and dependency confusion, and cannot dynamically respond to data and model changes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the deployable pipeline generation method provided by the above-mentioned embodiment, and will not be repeated here.

[0150] The present application also provides a computer program product, including a computer program, which implements the steps of the deployable pipeline generation method as described above when executed by a processor.

[0151] The computer program product provided by the present application can solve the technical problems that manual configuration and reliance on workflow tools can easily lead to configuration errors and dependency confusion, and cannot dynamically respond to data and model changes. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the deployable pipeline generation method provided by the above embodiment, and will not be repeated here.

[0152] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating a deployable pipeline, characterized in that: The method comprises the following steps: Using a directed acyclic graph modeling tool to create a plurality of workflow nodes, and obtaining a workflow by configuring the plurality of workflow nodes, the plurality of workflow nodes including operation types, data types, and input and output parameters; Applying the model evaluation analysis strategy and the model building algorithm to the training data set to obtain multiple model results, and selecting a deployment model by comparing the multiple model results; Dependencies of the deployment model are obtained from the workflow, and a deployable pipeline is generated according to the dependencies.

2. The deployable pipeline generation method according to claim 1, characterized in that: The step of obtaining the dependency of the deployment model from the workflow and generating a deployable pipeline according to the dependency includes: According to the topological sorting of the directed acyclic graph, traversing from the root node of the workflow, and obtaining the traversal result by recording the list of predecessor nodes and the operation type of each node; Generate a dependency tree structure of the workflow according to the traversal result; Identifying the dependency relationship of the deployment model from the dependency tree structure, and performing integrity verification on the dependency relationship to obtain a verification result; The dependency is updated according to the verification result and the dependency optimization information, and a deployable pipeline is generated according to the dependency.

3. The deployable pipeline generation method according to claim 2, characterized in that: The step of updating the dependency relationship according to the verification result and the dependency optimization information, and generating a deployable pipeline according to the dependency relationship includes: Update the dependency relationship according to the verification result and the dependency optimization information; Traversing the deployment model according to the dependency relationship to generate a node sequence of the deployment model; Obtain each node processing unit by recording each node information of the deployment model, and encapsulate each node processing unit into each node processing module; According to the order of the node sequence, the node processing modules are assembled into an initial deployable pipeline to generate a deployable pipeline.

4. The method for generating a deployable pipeline according to any one of claims 1 to 3, characterized in that: After the step of obtaining the dependency of the deployment model from the workflow and generating a deployable pipeline according to the dependency, the method further includes: When receiving a model deployment request, build the operating environment on the deployment platform; Based on the operating environment, the deployable pipeline is received, and according to the node sequence of the deployable pipeline, the configuration and parameters of each node processing module are parsed to obtain a parsing result; According to the analysis result and the operation instruction script, execute each node of the deployable pipeline in sequence; When executing the target model of the model deployment request, the output result is obtained by calling the target model.

5. The deployable pipeline generation method according to claim 4, characterized in that: After the step of obtaining the output result by calling the target model when executing the target model of the model deployment request, the step further includes: Comparing the initial output result with the expected result to obtain a comparison result; When the comparison result is inconsistent, the target model is adjusted and called again to obtain a deployment result; The deployment result is saved to a specified location, and a deployment log and a deployment execution report are generated; The workflow is parsed and a new deployable pipeline is generated, and the new deployable pipeline is verified by comparing the deployable pipeline with the new deployable pipeline.

6. The method for generating a deployable pipeline according to any one of claims 1 to 3, characterized in that: The step of creating a plurality of workflow nodes using a directed acyclic graph modeling tool and obtaining a workflow by configuring the plurality of workflow nodes comprises: Creating a plurality of workflow nodes by using a directed acyclic graph modeling tool, and configuring parameters of the plurality of workflow nodes; The dependency relationship of the plurality of workflow nodes is configured by a graphical method to form a directed acyclic graph structure, and the directed acyclic graph structure is used as the workflow.

7. The method for generating a deployable pipeline according to any one of claims 1 to 3, characterized in that: After the step of obtaining the dependency of the deployment model from the workflow and generating a deployable pipeline according to the dependency, the method further includes: According to a preset output format, serializing the relevant information of the deployable pipeline into a character string in the preset output format; The deployable pipeline is output, and a version number is assigned to the output deployable pipeline.

8. A deployable pipeline generation device, characterized in that: The deployable pipeline generating device comprises: A workflow creation module, used to create multiple workflow nodes using a directed acyclic graph modeling tool, and obtain a workflow by configuring the multiple workflow nodes, wherein the multiple workflow nodes include operation types, data types, and input and output parameters; A deployment model selection module is used to apply the model evaluation and analysis strategy and the model building algorithm to the training data set to obtain multiple model results, and select a deployment model by comparing the multiple model results; The deployable pipeline generation module is used to obtain the dependency relationship of the deployment model from the workflow and generate a deployable pipeline according to the dependency relationship.

9. A deployable pipeline generation device, characterized in that: The deployable pipeline generation device includes: a memory, a processor, and a deployable pipeline generation program stored in the memory and executable on the processor. When the deployable pipeline generation program is executed by the processor, the deployable pipeline generation method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a deployable pipeline generation program, and when the deployable pipeline generation program is executed by the processor, the deployable pipeline generation method according to any one of claims 1 to 7 is implemented.

Images