Algorithm library management system supporting visual arrangement of algorithm model construction process

Abstract

The invention discloses an algorithm library management system supporting visual arrangement of an algorithm model construction process. The algorithm library management system comprises an algorithm component management module, an algorithm component configuration module, an algorithm process arrangement module, an algorithm process operation module and an algorithm process log management module. The algorithm component management module is used for constructing and managing a full-process model algorithm component; the algorithm component configuration module is used for responding to a request of a user for setting specific parameters for each algorithm component in an algorithm model construction process; the algorithm process arrangement module provides a visual arrangement interaction mode, responds to the operation that a user drags algorithm components into a canvas, and connects the algorithm components according to a logic sequence through connecting lines to achieve data communication and transmission between the algorithm components. The algorithm process operation module is used for automatically executing algorithm components and resource scheduling according to the arranged logic sequence and completing execution of the whole algorithm model construction process; according to the method, a developer can construct and manage the algorithm model more efficiently and flexibly.

Description

Technical Field

[0001] The present invention relates to the field of the Internet, and in particular to an algorithm library management system that supports visual arrangement of algorithm model building processes. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, algorithm models are increasingly used in all walks of life. The construction of algorithm models usually involves multiple steps, including data preprocessing, feature engineering, model selection, training, evaluation, and deployment. However, the existing algorithm development process faces several major challenges:

[0003] 1. Process complexity: The algorithm development process consists of multiple stages, each of which requires different tools and techniques. This leads to the complexity of process management and increases development time and cost.

[0004] 2. Repetitive work: When building different algorithm models, developers often need to repeat similar tasks, such as data cleaning and feature selection, which reduces development efficiency.

[0005] 3. Difficulty in collaboration: When a team collaborates to develop an algorithm model, due to the lack of effective process management and version control, collaboration between team members becomes difficult, and work duplication or conflicts are prone to occur. Summary of the invention

[0006] In order to solve the above problems, the present invention provides an algorithm library management system that supports visual orchestration of algorithm model building process. The system simplifies the algorithm development process, improves development efficiency, promotes team collaboration, and optimizes resource allocation by providing an integrated environment.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] An algorithm library management system that supports visual orchestration of algorithm model building processes, including an algorithm component management module, an algorithm component configuration module, an algorithm process orchestration module, an algorithm process running module, and an algorithm process log management module;

[0009] The algorithm component management module is used to build and manage the full-process model algorithm components from data processing, feature engineering to algorithm model;

[0010] The algorithm component configuration module is used to respond to a user's request to set specific parameters for each algorithm component in the algorithm model building process;

[0011] The algorithm process arrangement module provides an interactive method of visual arrangement, responds to the user's operation of dragging the algorithm component into the canvas, and connects different algorithm components in a logical order on the canvas to realize data connection and transmission between multiple algorithm components;

[0012] The algorithm process running module is used to automatically execute algorithm components and perform resource scheduling according to the arranged logical sequence to complete the execution of the entire algorithm model building process;

[0013] The algorithm process log management module is used to record the detailed execution status of each component in the entire algorithm model building process and provide output results for download.

[0014] Furthermore, the algorithm component management module includes a data access module, a data preprocessing module, a feature engineering module, an algorithm modeling module, a model evaluation module and a data visualization module;

[0015] The data access module supports multiple database connections and multiple file format uploads;

[0016] The data preprocessing module provides a variety of data preprocessing components including a data splitting component, a data cleaning component, a data screening component, and a data completion component;

[0017] The feature engineering module provides a variety of data processing and conversion components including a data standardization component, a normalization component, a data filtering algorithm component, and a principal component analysis component;

[0018] The algorithm modeling module includes a variety of algorithm modeling components including a regression algorithm component, a classification algorithm component and a clustering algorithm;

[0019] The model evaluation module provides components for calculating multiple model evaluation indicators, including an accuracy evaluation component, a recall evaluation component, an F1 score evaluation component, and a mean square error evaluation component;

[0020] The data visualization module provides a variety of visualization graphic components including pie charts, line charts, histograms, scatter plots, heat maps, and candlestick charts, which are used to display the prediction results of the algorithm model in the form of graphs.

[0021] Furthermore, the algorithm process orchestration module includes a visual orchestration interactive interface, a process editor and a process validator;

[0022] The visual arrangement interactive interface is a visual canvas for displaying the algorithm flow, and displays the algorithm flow in a graphical manner;

[0023] The process editor is used to respond to the user's drag-and-drop operation to arrange the algorithm process, and realize data connectivity and transmission between multiple algorithm components through connections; the connection represents the data flow;

[0024] The process validator is used to check the correctness and completeness of the orchestration process, verify the consistency of the process, and verify whether the parameter configuration of each component is valid, including data types and dependencies.

[0025] Furthermore, in the algorithm component configuration module, the specific parameters set by the user for the algorithm component include input parameters, output parameters, algorithm model hyperparameters, and mirror configuration of execution logic;

[0026] The input parameters and output parameters include the data type and name of the data stream;

[0027] The algorithm model hyperparameters are parameters set before the algorithm model learning process begins;

[0028] The image of the execution logic is a Docker image of a script or code logic executed based on the algorithm component.

[0029] Furthermore, the algorithm process running module includes a process execution engine, a resource allocator, a state monitoring module and an execution result processing module;

[0030] The process execution engine is used to parse and automatically execute the algorithm execution logic that has been choreographed by the user;

[0031] The resource allocator is used to dynamically allocate computing resources according to the resource requirements of the algorithm tasks and the system resource status;

[0032] The status monitoring module is used to track the execution progress, performance indicators and status changes of algorithm tasks in real time;

[0033] The execution result processing module is used to collect the execution results of the algorithm tasks and classify and store the execution results in different types of storage systems; among them, structured data is stored in a relational database, while unstructured data is stored in an object storage device, and the execution log information is stored in a message queue.

[0034] Furthermore, the algorithm process log management module includes a log collection module, a log storage module, a log analysis module and a log viewing module;

[0035] The log collection module is used to collect the execution status information of each algorithm component in the algorithm flow, including the execution status, output results, execution time, reasons for execution failure, and result information output after successful execution;

[0036] The log storage module is used to store the information collected by the log collection module in a database so as to permanently save the execution status information of the algorithm component and query log information;

[0037] The log analysis module is used to identify abnormal patterns in the execution of algorithm components and output the reasons for abnormal execution of algorithm components;

[0038] The log viewing module is used to display log information and analysis results to users, so that users can view the details of algorithm process execution.

[0039] Further, the regression algorithm components include decision tree regression, gradient boosted decision tree regression, linear regression, random forest regression, linear support vector machine regression, multilayer perceptron regression, histogram-based gradient boosted decision tree regression, kernel ridge regression, K-nearest neighbor regression, radius neighbor regression, Bayesian ridge regression, elastic net regression;

[0040] The classification algorithm components include linear support vector machine classification, Bernoulli constrained Boltzmann machine classification, multi-layer perceptron classification, K nearest neighbor classification, radius neighbor classification, multi-output classification, support vector machine classification, logistic regression, logistic regression with cross-validation, passive attack classification and perceptron algorithms;

[0041] The clustering algorithms include agglomerative hierarchical clustering, Birch clustering, density-based clustering with noise, mean shift clustering, spectral clustering, K-means clustering, feature fusion clustering, and mini-batch K-means clustering.

[0042] Furthermore, the algorithm library management system also includes a Kubernetes container orchestration system, an object storage server, a database, and a message queue;

[0043] The Kubernetes container orchestration system is used to manage the life cycle of containers, including creation, scheduling, expansion, and destruction, providing a scalable operating environment for algorithm components and ensuring efficient use of resources;

[0044] The object storage server is used to store unstructured data;

[0045] The database is used to store structured data;

[0046] The message queue is used to store the status and results of the algorithm process execution.

[0047] Furthermore, the specific steps of the process execution engine parsing and automatically executing the algorithm execution logic that has been choreographed by the user are as follows:

[0048] Step 1: According to the image information configured in the algorithm component, pull the corresponding Docker image from the image repository, and then create a new Docker container based on the Docker image;

[0049] Step 2: Run the script or program of the algorithm component in the Docker container;

[0050] Step three: Push the algorithm execution status and results after the algorithm component is executed to the message array; the algorithm process log management module subscribes to and pulls the messages of the algorithm component execution status results from the message queue, processes the messages, and finally permanently saves the execution status of the algorithm component in the database for historical recording and subsequent analysis.

[0051] A visually arranged algorithm model construction method is implemented based on an algorithm library management system that supports visual arrangement of the algorithm model construction process, and is characterized in that the method comprises the following steps:

[0052] S001: responding to a user's request to create a new algorithm orchestration process project and opening a new canvas;

[0053] S002: responding to the user's request to select a full-process model tool component from data processing, feature engineering to algorithm model, and displaying the corresponding algorithm component in the canvas;

[0054] S003: responding to the user's request for specific parameter settings for each algorithm component and mirror configuration of the execution logic;

[0055] S004: responding to the user's arrangement of all algorithm components in the canvas in the order of algorithm business requirements, and connecting the algorithm components with lines representing data flows;

[0056] S005: Respond to the user's execution instruction for the completed algorithm flow.

[0057] S006: Respond to the user's request to view the execution results of the algorithm process.

[0058] The beneficial effects of the present invention are as follows:

[0059] Integrated development environment: The algorithm library management system provided by the present invention integrates the tools and resources required for algorithm development, thereby simplifying the development process and lowering the application threshold of the algorithm model.

[0060] Process orchestration capability: The present invention allows users to define and automate various stages of algorithm development, thereby reducing repetitive work and improving development efficiency.

[0061] Team collaboration support: The present invention supports multi-user collaboration through version control and task allocation mechanisms to ensure coordinated work.

[0062] Through the present invention, algorithm developers can build and manage algorithm models more efficiently and flexibly, which also lowers the application threshold of algorithm models, while reducing development costs and improving the performance of algorithm model construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the functional modules of an algorithm library management system that supports visual arrangement of algorithm model building process according to an embodiment of the present invention.

[0064] Figure 2 It is a structural schematic diagram of an algorithm library management device that supports visual arrangement of algorithm model construction process in an embodiment of the present invention.

[0065] Figure 3 It is a flowchart of a method for constructing a visually arranged algorithm model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0067] like Figure 1 As shown, the algorithm library management system supporting the visual arrangement of the algorithm model construction process according to the embodiment of the present invention includes:

[0068] The algorithm component management module 100 is developed based on a machine learning framework and is used to build and manage full-process model algorithm components from data processing, feature engineering to algorithm models.

[0069] The algorithm component configuration module 101 is used to respond to the user's request to set specific parameters for each algorithm component in the algorithm model building process. The specific parameters include the input parameters, output parameters and algorithm model hyperparameters of the algorithm component.

[0070] The algorithm process arrangement module 102 provides an interactive method of visual arrangement, responds to the user's operation of dragging the algorithm component into the canvas, and connects different algorithm components in a logical order on the canvas to realize data connection and transmission between multiple algorithm components.

[0071] The algorithm process running module 103 automatically executes the algorithm components and performs resource scheduling according to the arranged sequence to complete the execution of the entire algorithm model building process.

[0072] The algorithm process log management module 104 is used to record the detailed execution of each component in the entire algorithm model building process and provide output results for download. Through the algorithm process log management module 104, users can view the execution status of each component in the entire algorithm model building process, including execution status, (in progress, success, failure, etc.) output results, reasons for execution failure, and execution time; output results include algorithm models, model prediction results, etc.

[0073] 1. Algorithm component management module

[0074] The algorithm component management module includes data access module, data preprocessing module, feature engineering module, algorithm modeling module, model evaluation module and data visualization module.

[0075] Among them, the data access module supports multiple database connections such as Mysql, Hive, Postgres, and multiple file uploads such as Excel, txt, csv, etc. Therefore, the data source access methods for algorithm model training are diverse.

[0076] The data preprocessing module provides a variety of data preprocessing components including data splitting components, data cleaning components, data screening components, and data completion components.

[0077] The feature engineering module provides a variety of data processing and conversion components, including data standardization components, normalization components, data filtering algorithm components, and principal component analysis components.

[0078] The algorithm modeling module includes a variety of algorithm modeling components, including regression algorithm components, classification algorithm components and clustering algorithms. Among them, the regression algorithm includes mainstream regression algorithms, such as decision tree regression, gradient boosting decision tree regression, linear regression, random forest regression, linear support vector machine regression, multilayer perceptron regression, histogram-based gradient boosting decision tree regression, kernel ridge regression, K nearest neighbor regression, radius neighbor regression, Bayesian ridge regression, elastic network regression and other algorithms; the classification algorithm includes mainstream classification algorithms, such as linear support vector machine classification, Bernoulli constrained Boltzmann machine classification, multilayer perceptron classification, K nearest neighbor classification, radius neighbor classification, multi-output classification, support vector machine classification, logistic regression, logistic regression with cross validation, passive attack classification, perceptron and other algorithms; the clustering algorithm includes mainstream clustering algorithms, such as agglomerative hierarchical clustering, Birch clustering, density-based clustering with noise, mean shift clustering, spectral clustering, K-means clustering, feature fusion clustering, small batch K-means clustering and other algorithms.

[0079] The model evaluation module provides components for calculating various model evaluation indicators, including accuracy evaluation component, recall evaluation component, F1 score evaluation component, and mean square error evaluation component.

[0080] The data visualization module provides a variety of visualization graphic components including pie charts, line charts, histograms, scatter plots, heat maps, and candlestick charts, which are used to display the prediction results of the algorithm model in the form of graphs.

[0081] 2. Algorithm component configuration module 101

[0082] The algorithm component configuration module 101 responds to the mirror configuration of input parameters, output parameters, algorithm hyperparameters and execution logic set by the user for each algorithm component, and executes the corresponding logic according to these configurations during the subsequent algorithm process execution.

[0083] The input parameters and output parameters of an algorithm component mainly refer to the data type and name of the data flow (input data and output data) of the algorithm component. The supported data types include string, number, Boolean value, Json, etc. Data transmission can only be completed when the data types of the node output of the upstream algorithm component and the node input of the downstream algorithm component are the same, otherwise there will be an error prompt when the algorithm flow is executed.

[0084] Algorithm hyperparameters refer to parameters set before the algorithm model learning process begins, which are different from parameters automatically learned during model training (i.e., model weights). The choice of hyperparameters has a significant impact on model performance, but they are not learned directly from data. In this system, the hyperparameters of algorithm components are exposed for users to set freely. For example, the parameters of the linear support vector machine regression algorithm component are epsilon (interval threshold), kernel (kernel function), max_iter (maximum number of iterations), tol (tolerance), dual (whether to use dual problem), and random_state (random state).

[0085] The execution logic image is a docker image based on the script or code logic executed by the algorithm component, which is stored in the docker warehouse. You can follow up on the needs to select the docker images corresponding to different versions of components (different execution logic). The use of docker warehouses and images is to facilitate version management of the same algorithm logic. For example, the first method of data preprocessing logic A is built into the image DataProcessingA: V1 (version 1), and the second method of data preprocessing logic A is built into the image DataProcessingA: V2 (version 2).

[0086] 3. Algorithm Process Arrangement Module 102

[0087] The algorithm process orchestration module 102 is to complete the entire algorithm process by arranging multiple algorithm components, including the entire algorithm process from data upload → data preprocessing → feature engineering → machine learning → result analysis → iterative optimization. The entire algorithm process can be flexibly built through the algorithm component orchestration method, lowering the threshold for technicians to use machine learning algorithms. The algorithm process orchestration module 102 includes a visual orchestration interactive interface, a process editor, and a process validator.

[0088] The visual orchestration interactive interface is a visual canvas that displays the algorithm flow. It is a flowchart display area that displays the algorithm flow in a graphical way.

[0089] The process editor is used to respond to the user's drag-and-drop operation to arrange the algorithm process and realize data connectivity and transmission between multiple algorithm components through connections. The connection between algorithm components represents the data flow.

[0090] The process validator is used to check the correctness and completeness of the orchestration process, verify the consistency of the process, and verify whether the parameter configuration of each component is valid, including data types and dependencies.

[0091] 4. Algorithm process operation module 103

[0092] The algorithm process running module 103 includes a process execution engine, a resource allocator, a status monitoring module and an execution result processing module.

[0093] The process execution engine is used to parse and automatically execute the algorithm execution logic that the user has already choreographed. It can automatically execute each algorithm step according to the established process arrangement, including the following steps:

[0094] Step 1: According to the image information configured in the algorithm component, pull the corresponding Docker image from the image repository, and then create a new Docker container based on the Docker image;

[0095] Step 2: Run the script or program of the algorithm component in the Docker container. For example, execute the core logic of the linear regression component, that is, load the training data, execute the training process, and optimize the model parameters.

[0096] Step 3: Push the algorithm execution status (such as success, failure, exception, etc.) and results (such as model performance indicators) after the algorithm component is executed to the message array; the algorithm process log management module subscribes to and pulls the messages of the algorithm component execution status results from the message queue, and processes the messages, and finally permanently saves the execution status of the algorithm component in the database for historical records and subsequent analysis.

[0097] The resource allocator is used to dynamically allocate computing resources according to the resource requirements of the algorithm tasks and the system resource status, thereby optimizing the allocation of computing resources.

[0098] The status monitoring module is used to track the execution progress, performance indicators and status changes of algorithm tasks in real time, thereby monitoring the execution status of the algorithm process.

[0099] The execution result processing module is used to collect the execution results of the algorithm tasks, and classify and store the execution results in different types of storage systems; among them, structured data (such as statistical indicators, performance evaluation results, etc.) are stored in relational databases, and unstructured data (such as training data files, algorithm model files) are stored in object storage. The execution status information (such as success, failure, etc.), execution time and other execution log information are stored in the message queue MQ system for subscription and analysis by the algorithm process log management module 104.

[0100] 5. Algorithm process log management module 104

[0101] The algorithm process log management module 104 includes a log collection module, a log storage module, a log analysis module and a log viewing module.

[0102] The log collection module is used to collect the execution status information of each algorithm component in the algorithm process, including execution status, output results, execution time, reasons for execution failure, output result information after successful execution, etc. This module mainly subscribes to and parses the log information of algorithm execution from the message queue to complete the log collection function.

[0103] The log storage module is used to store the information collected by the log collection module in a database so as to permanently save the execution status information of the algorithm component and query log information.

[0104] The log analysis module is used to identify abnormal patterns in the execution of algorithm components and output the reasons for abnormal execution of algorithm components.

[0105] The log viewing module is used to display log information and analysis results to users, so that users can view the details of algorithm process execution.

[0106] like Figure 2 As shown, the algorithm library management system of this embodiment that supports visual orchestration of the algorithm model building process, in addition to the business function modules such as the algorithm component management module, the algorithm component configuration module, the algorithm process orchestration module, the algorithm process running module, and the algorithm process log management module, also includes a Kubernetes container orchestration system, an object storage server, a database, and a message queue.

[0107] Among them, the Kubernetes container orchestration system is used to manage the life cycle of containers, including creation, scheduling, expansion and destruction, providing a scalable operating environment for algorithm components and ensuring efficient use of resources. Kubernetes provides powerful container orchestration capabilities that can automate the deployment, expansion and management of containers, and supports horizontal expansion, which can automatically adjust the number of container instances according to resource requirements and load. When executing the algorithm process, the algorithm process execution engine will create a new docker container for each algorithm component in the algorithm process, configure the corresponding environment variables, and then execute the core logic of the algorithm component. Use the Kubernetes system to dynamically allocate resources to complete task calculations in the algorithm process. For example, the logistic regression algorithm component requires a large amount of computing power resources for model training.

[0108] The object storage server is a storage system for storing unstructured data. In this embodiment, it is responsible for storing files such as algorithm training data (such as csv and excel files storing training data) and algorithm models involved in the algorithm process.

[0109] The database is used to store structured data, such as component configuration information and log information in the algorithm process.

[0110] The message queue is used to store information such as the status and results of the algorithm process execution. The message queue can be used to monitor the execution of the algorithm process in real time and is used to transmit messages and data between algorithm components.

[0111] like Figure 3 As shown, another embodiment of the present invention provides a visually arranged algorithm model construction method, which is implemented based on an algorithm library management system that supports visual arrangement of the algorithm model construction process, and the method includes the following steps:

[0112] S001: responding to a user's request to create a new algorithm orchestration process project and opening a new canvas;

[0113] S002: Respond to the user's request to select the full-process model tool component from data processing, feature engineering to algorithm model, and display the corresponding algorithm component in the canvas. Specifically:

[0114] (1) Data processing

[0115] The algorithm model construction first needs to obtain training data. Therefore, it is first necessary to respond to the user's selection and drag of the data access component, such as supporting multiple database connection components such as Mysql, Hive, Postgres, or supporting multiple file data uploads such as Excel, txt, csv, etc. Data access components. Then, respond to the user's selection and drag of the data preprocessing function component and the feature engineering function component, and display the component in the canvas. In the algorithm model training, data preprocessing is a crucial step, which plays a decisive role in improving the performance and accuracy of the algorithm. The algorithm component management module in the system of the present invention provides a data preprocessing component for processing functions such as data splitting, data cleaning, data screening, and data completion. According to business needs, respond to the user's selection and drag, put the corresponding component into the canvas, and perform tasks related to data preprocessing. Secondly, in the algorithm model training, feature engineering is also a key link, which involves the process of extracting, building and selecting features from raw data to improve the performance and accuracy of the model. Also according to business needs, respond to the user's selection and drag, and put the required feature engineering function component into the canvas. Thirdly, in algorithm model training, the data set needs to be divided into training set, validation set and test set, usually using methods such as 70-30, 80-20 or cross validation. In response to the user's selection and dragging, the data set partitioning component is placed in the canvas, and the user configures the data set partitioning ratio.

[0116] (2) Feature Engineering

[0117] In algorithm model training, feature engineering is also a key link, which involves the process of extracting, building and selecting features from raw data to improve the performance and accuracy of the model. Therefore, in response to the user's selection and dragging of feature engineering function components, the feature engineering function components are placed in the canvas.

[0118] (3) Algorithm Modeling

[0119] Select the appropriate algorithm model according to the type of problem the algorithm needs to solve and the data characteristics, such as selecting linear regression, decision tree, random forest regression and other algorithm models to solve a regression algorithm problem. Therefore, the algorithm modeling component that responds to user selection and dragging is used for model training.

[0120] (4) Model evaluation

[0121] In the algorithm model building process, the validation set is needed to evaluate the performance of the model. Common evaluation indicators include accuracy, recall, F1 score, mean square error, etc. Therefore, the accuracy evaluation component, recall evaluation component, mean square error evaluation component, etc. that respond to user selection and dragging are placed in the canvas for algorithm model evaluation.

[0122] (6) Visualization

[0123] This step is optional. The data visualization function module in the system of the present invention provides common visualization graphic components, such as pie charts, line charts, histograms, scatter plots, heat maps, candlestick charts, etc., so as to display the prediction results of the algorithm model in the form of a graph. Therefore, the data visualization component selected and dragged by the user is responded to and placed in the canvas.

[0124] S003: Respond to the user's request for specific parameter settings for each algorithm component and execute the mirror configuration of the logic.

[0125] Configure each component placed in the canvas (included in the algorithm flow) according to the problem requirements, and configure its input parameters, output parameters, algorithm hyperparameters, and image configuration of the execution logic. For example, the hyperparameters of the linear support vector machine regression algorithm component include epsilon (interval threshold), kernel (kernel function), max_iter (maximum number of iterations), tol (tolerance), dual (whether to use dual problems), and random_state (random state). The image of the execution logic will have the latest image version selected by default.

[0126] S004: Responding to the user's arrangement of all algorithm components in the canvas in the order of algorithm business requirements, and connecting the algorithm components with lines representing data flows.

[0127] The canvas is a user-visualized interactive interface for arranging algorithm components. Users can adjust the algorithm flow according to their needs in this canvas, such as adding, deleting, or rearranging algorithm components. Lines are used to connect algorithm components on the canvas to represent data flow, ensuring that data flows from the input source to the preprocessing component, then to feature extraction, and then to model training and model evaluation. It is necessary to ensure that the data flow is correctly transmitted between components, and the data type, size, and format of the components need to be configured to ensure that the output of the previous component is the input of the next component.

[0128] S005: Respond to the user's execution instruction for the completed algorithm flow.

[0129] In response to the user's execution request, the algorithm flow execution engine begins to parse and execute the algorithm flow that has been arranged in the user interface, and then automatically executes each algorithm step according to the established flow arrangement. The resource allocator can dynamically allocate computing resources according to the resource requirements of the algorithm task and the system resource status. During the execution of the algorithm flow, the algorithm execution status and results will be collected and stored in the message queue MQ (Message Queue) system.

[0130] S006: Respond to the user's request to view the execution results of the algorithm process.

[0131] The algorithm flow log management module can collect the execution status information of the algorithm flow. The execution status change information includes but is not limited to the execution status (executing, execution failure, execution success), the reason for execution failure, and the result information output after successful execution. Therefore, you can view the detailed execution status and result information in this module. You can also download the output results of each component, for example, you can download the output results of the algorithm modeling component (model file, generally a file with the file suffix .model).

[0132] The algorithm library management system and orchestration method provided in this embodiment integrate the tools and resources required for algorithm development, thereby simplifying the development process, reducing development costs, and reducing the manpower and time costs in the development process through automation and collaborative toolboxes; at the same time, it also lowers the application threshold of algorithm models, and enables non-professional users to participate in algorithm development and application by providing an easy-to-use algorithm orchestration interface and algorithm toolbox, thereby simplifying the construction and management process of algorithm models.

[0133] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. An algorithm library management system that supports visual arrangement of algorithm model building process, characterized in that: It includes algorithm component management module, algorithm component configuration module, algorithm process orchestration module, algorithm process operation module, and algorithm process log management module; The algorithm component management module is used to build and manage the full-process model algorithm components from data processing, feature engineering to algorithm model; The algorithm component configuration module is used to respond to a user's request to set specific parameters for each algorithm component in the algorithm model building process; The algorithm process arrangement module provides an interactive method of visual arrangement, responds to the user's operation of dragging the algorithm component into the canvas, and connects different algorithm components in a logical order on the canvas to realize data connection and transmission between multiple algorithm components; The algorithm process running module is used to automatically execute algorithm components and perform resource scheduling according to the arranged logical sequence to complete the execution of the entire algorithm model building process; The algorithm process log management module is used to record the detailed execution status of each component in the entire algorithm model building process and provide output results for download.

2. The algorithm library management system supporting the visual arrangement of algorithm model building process according to claim 1 is characterized in that: The algorithm component management module includes a data access module, a data preprocessing module, a feature engineering module, an algorithm modeling module, a model evaluation module and a data visualization module; The data access module supports multiple database connections and multiple file format uploads; The data preprocessing module provides a variety of data preprocessing components including a data splitting component, a data cleaning component, a data screening component, and a data completion component; The feature engineering module provides a variety of data processing and conversion components including a data standardization component, a normalization component, a data filtering algorithm component, and a principal component analysis component; The algorithm modeling module includes a variety of algorithm modeling components including a regression algorithm component, a classification algorithm component and a clustering algorithm; The model evaluation module provides components for calculating multiple model evaluation indicators, including an accuracy evaluation component, a recall evaluation component, an F1 score evaluation component, and a mean square error evaluation component; The data visualization module provides a variety of visualization graphic components including pie charts, line charts, histograms, scatter plots, heat maps, and candlestick charts, which are used to display the prediction results of the algorithm model in the form of graphs.

3. The algorithm library management system supporting the visual arrangement of algorithm model building process according to claim 2 is characterized in that: The algorithm process orchestration module includes a visual orchestration interactive interface, a process editor and a process validator; The visual arrangement interactive interface is a visual canvas for displaying the algorithm flow, and displays the algorithm flow in a graphical manner; The process editor is used to respond to the user's drag-and-drop operation to arrange the algorithm process, and realize data connectivity and transmission between multiple algorithm components through connections; the connection represents the data flow; The process validator is used to check the correctness and completeness of the orchestration process, verify the consistency of the process, and verify whether the parameter configuration of each component is valid, including data types and dependencies.

4. The algorithm library management system supporting the visual arrangement of algorithm model building process according to claim 1 is characterized in that: In the algorithm component configuration module, the specific parameters set by the user for the algorithm component include input parameters, output parameters, algorithm model hyperparameters, and mirror configuration of execution logic; The input parameters and output parameters include the data type and name of the data stream; The algorithm model hyperparameters are parameters set before the algorithm model learning process begins; The image of the execution logic is a Docker image of a script or code logic executed based on the algorithm component.

5. The algorithm library management system supporting the visual arrangement of algorithm model building process according to claim 1 is characterized in that: The algorithm process running module includes a process execution engine, a resource allocator, a status monitoring module and an execution result processing module; The process execution engine is used to parse and automatically execute the algorithm execution logic that has been choreographed by the user; The resource allocator is used to dynamically allocate computing resources according to the resource requirements of the algorithm tasks and the system resource status; The status monitoring module is used to track the execution progress, performance indicators and status changes of algorithm tasks in real time; The execution result processing module is used to collect the execution results of the algorithm tasks and classify and store the execution results in different types of storage systems; among them, structured data is stored in a relational database, while unstructured data is stored in an object storage device, and the execution log information is stored in a message queue.

6. The algorithm library management system supporting the visual arrangement of algorithm model building process according to claim 1 is characterized in that: The algorithm process log management module includes a log collection module, a log storage module, a log analysis module and a log viewing module; The log collection module is used to collect the execution status information of each algorithm component in the algorithm flow, including the execution status, output results, execution time, reasons for execution failure, and result information output after successful execution; The log storage module is used to store the information collected by the log collection module in a database so as to permanently save the execution status information of the algorithm component and query log information; The log analysis module is used to identify abnormal patterns in the execution of algorithm components and output the reasons for abnormal execution of algorithm components; The log viewing module is used to display log information and analysis results to users, so that users can view the details of algorithm process execution.

7. The algorithm library management system supporting the visual arrangement of algorithm model building process according to claim 2 is characterized in that: The regression algorithm components include decision tree regression, gradient boosted decision tree regression, linear regression, random forest regression, linear support vector machine regression, multilayer perceptron regression, histogram-based gradient boosted decision tree regression, kernel ridge regression, K nearest neighbor regression, radius neighbor regression, Bayesian ridge regression, and elastic net regression; The classification algorithm components include linear support vector machine classification, Bernoulli constrained Boltzmann machine classification, multi-layer perceptron classification, K nearest neighbor classification, radius neighbor classification, multi-output classification, support vector machine classification, logistic regression, logistic regression with cross-validation, passive attack classification and perceptron algorithms; The clustering algorithms include agglomerative hierarchical clustering, Birch clustering, density-based clustering with noise, mean shift clustering, spectral clustering, K-means clustering, feature fusion clustering, and mini-batch K-means clustering.

8. The algorithm library management system supporting the visual arrangement of algorithm model building process according to claim 1 is characterized in that: The algorithm library management system also includes a Kubernetes container orchestration system, an object storage server, a database, and a message queue; The Kubernetes container orchestration system is used to manage the life cycle of containers, including creation, scheduling, expansion, and destruction, providing a scalable operating environment for algorithm components and ensuring efficient use of resources; The object storage server is used to store unstructured data; The database is used to store structured data; The message queue is used to store the status and results of the algorithm process execution.

9. The algorithm library management system supporting visual arrangement of algorithm model building process according to claim 5, characterized in that: The specific steps of the process execution engine parsing and automatically executing the algorithm execution logic that has been choreographed by the user are as follows: Step 1: According to the image information configured in the algorithm component, pull the corresponding Docker image from the image repository, and then create a new Docker container based on the Docker image; Step 2: Run the script or program of the algorithm component in the Docker container; Step 3: Push the algorithm execution status and results after the algorithm component is executed to the message array; The algorithm process log management module subscribes to and pulls messages of algorithm component execution status results from the message queue, processes the messages, and finally permanently stores the execution status of the algorithm component in the database for historical recording and subsequent analysis.

10. A visually arranged algorithm model construction method, the method is implemented based on the algorithm library management system supporting visual arrangement of algorithm model construction process as claimed in any one of claims 1 to 9, characterized in that: The method comprises the following steps: S001: responding to a user's request to create a new algorithm orchestration process project and opening a new canvas; S002: responding to the user's request to select a full-process model tool component from data processing, feature engineering to algorithm model, and displaying the corresponding algorithm component in the canvas; S003: responding to the user's request for specific parameter settings for each algorithm component and mirror configuration of the execution logic; S004: responding to the user's arrangement of all algorithm components in the canvas in the order of algorithm business requirements, and connecting the algorithm components with lines representing data flows; S005: Respond to the user's execution instruction for the completed algorithm flow. S006: Respond to the user's request to view the execution results of the algorithm process.

Images