A unified encapsulation method and system of data mining algorithms

Abstract

This invention belongs to the field of data processing technology and discloses a unified encapsulation method and system for data mining algorithms. The method includes encapsulating data mining algorithms into algorithm components that match a common interface template, building an evaluation model based on the algorithm components, constructing a pipeline using the algorithm components through a graphical interface to obtain the data mining model, automatically adjusting the data mining model through machine learning and optimization algorithms, and finally encapsulating and deploying the data mining model to a target area. This invention enables users to call different algorithms through a unified interface. Calling the algorithms as a service makes the algorithm components more adaptable, facilitates the maintenance and expansion of data mining software, lowers the barrier to entry for data mining, improves the interactive operation and user experience of machine learning, enhances the flexibility and maintainability of data mining, reduces reliance on business and domain knowledge, saves time and resources, and improves model performance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a unified encapsulation method and system for data mining algorithms. Background Technology

[0002] The rapid growth of big data has presented enormous challenges to data processing and analysis in specialized fields such as equipment and combat applications. Data mining, a technique for discovering hidden patterns and correlations in large-scale data, is of great significance for solving practical problems, improving decision-making efficiency, and optimizing resource allocation. Data mining is the process of searching for hidden information within massive amounts of data using algorithms. It extracts valuable information and discovers patterns from diverse data. Methods employed include statistics, online analytical processing, intelligence retrieval, machine learning, expert systems, and pattern recognition. Data mining algorithm encapsulation is a technology and method that encapsulates algorithms for each stage of the data mining process—business understanding, data understanding, data preparation, data modeling, model evaluation, and deployment—using a unified interface to form a series of algorithm components.

[0003] However, challenges and bottlenecks remain in the application of data mining algorithms, limiting their widespread application and in-depth development. For example, professional fields such as equipment and combat applications generate massive amounts of data daily, including structured data (such as data in databases), semi-structured data (such as log files and XML files), and unstructured data (such as text, images, and audio). This data comes from various sources, with different formats, types, and qualities, posing significant challenges to data mining. Data mining algorithms are diverse, including classification, regression, clustering, and association rule mining. Different algorithms are suitable for different data types and problem scenarios. Model integration and deployment are often complex and cumbersome, involving collaborative work between models, parameter passing, and result integration, requiring substantial human and time resources. Furthermore, data mining application environments are diverse, covering different professional fields and business scenarios. Different application scenarios have different requirements for data mining tools and technologies, necessitating customized solutions to meet specific business needs.

[0004] Therefore, how to provide a unified encapsulation method and system for data mining algorithms is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a unified encapsulation method and system for data mining algorithms to solve the problems in the prior art.

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0007] According to a first aspect of the present invention, a unified encapsulation method for data mining algorithms is provided.

[0008] In one embodiment, the unified encapsulation method for the data mining algorithm includes: Collect at least one data mining algorithm, analyze the data mining algorithm, construct a general interface template for the data mining algorithm based on the parameters obtained after analysis, encapsulate the data mining algorithm into algorithm components that match the general interface template, and build an evaluation model based on the algorithm components. Calculate all algorithm components to obtain component metadata files, and store the algorithm components and their metadata files in the target area. Use the algorithm components to build a pipeline through a graphical interface, define the execution order and dependencies of the algorithm components, obtain the data mining model, evaluate the model, and output an evaluation report of the data mining model. Based on a pre-configured data mining model, the data mining model is automatically adjusted through machine learning and optimization algorithms, and an evaluation model is invoked to evaluate the adjusted data mining model. Upload at least one well-tuned data mining model to the model repository, input the parameters of the data mining model, package and deploy the data mining model to the target region, and analyze the deployment strategy and automated deployment process of the data mining model in the target region.

[0009] In one embodiment, at least one data mining algorithm is collected and analyzed. Based on the parameters obtained after analysis, a general interface template for the data mining algorithm is constructed. The data mining algorithm is encapsulated into algorithm components that match the general interface template. An evaluation model is then constructed based on the algorithm components, including: Analyze the inputs, outputs, and dependencies of data mining algorithms to identify common requirements and differences among them. Build basic and extended interfaces, unify the input and output of the interfaces, and generate algorithm metadata files for data mining algorithms; The data mining algorithm is refactored to match its interface, and the running environment of the data mining algorithm is isolated using virtual environment or containerization technology to obtain the algorithm component.

[0010] In one embodiment, a basic interface and an extended interface are constructed, the input and output of the interface are unified, and an algorithm metadata file is generated for the data mining algorithm, including: The basic interface includes preset necessary processing methods, while the extended interface is set based on the data mining algorithm type; The interface input is uniformly structured data objects and parameter configuration files, and the interface output is uniformly result objects.

[0011] In one embodiment, building an evaluation model based on algorithmic components includes: The evaluation model is built based on pre-configured accuracy, computational efficiency, K-fold cross-validation, independent test set validation, learning curve analysis, regularization, and early stopping.

[0012] In one embodiment, the evaluation model is constructed based on pre-configured accuracy, computational efficiency, K-fold cross-validation, independent test set validation, learning curve analysis, and regularization and early stopping, including: Accuracy assessments include accuracy, precision, F1 score, ROC curve, AUC score, and regression task. Accuracy is assessed based on accuracy, precision, F1 score, ROC curve and AUC value, and regression task. Data is collected by recording training time, inference time, monitoring resource consumption and computational efficiency, and combining this with monitoring tools.

[0013] In one embodiment, computation is performed on all algorithm components to obtain component metadata files. The algorithm components and their metadata files are then stored in a target area. A pipeline is constructed using the algorithm components through a graphical interface, defining the execution order and dependencies of the algorithm components to obtain a data mining model. An evaluation report of the data mining model is then output, including: The algorithm components are moved from the storage area to the graphical interface, and the algorithm components are connected according to the preset logic to form an algorithm component pipeline. The parameters of the algorithm components are set, the compatibility between the algorithm components is checked, and the data mining model is obtained. The pipeline is executed automatically, and the status of the algorithm components is monitored and recorded in real time.

[0014] In one embodiment, the process of automatically adjusting the data mining model based on a pre-configured model using machine learning and optimization algorithms, and then evaluating the adjusted model by invoking an evaluation model, includes: The raw data is preprocessed, encapsulated into components using a common interface template, and the component characteristics are described using metadata. Based on domain knowledge, we automatically construct derived features, filter out key features, and train candidate data mining models according to the data scale and the complexity of key features. Define the hyperparameter range of the data mining algorithm model, and use optimization algorithms to adjust the data mining algorithm model.

[0015] In one embodiment, at least one tuned data mining model is uploaded to a model repository, the parameters of the data mining model are input, the data mining model is packaged and deployed to the target region, and the deployment strategy and automated deployment process of the data mining model in the target region are analyzed, including: Upload the pre-trained data mining model file, verify the format and integrity of the data mining model file, input the data mining model parameters, and automatically generate the version number; Based on a pre-selected deployment region, the data mining model is packaged, and an image file of the data mining model is built by calling CI / CD tools and pushed. Based on the built image file, it is deployed via Kubernetes. Based on the characteristics of the data mining model and the server status, the optimal solution and resource allocation scheme for the data mining model on the server are calculated, and automatic and flexible deployment is performed.

[0016] In one embodiment, based on the characteristics of the data mining model and the server status, calculating the optimal solution for server deployment and resource allocation scheme of the data mining model, and performing automatic and flexible deployment includes: The optimization objective is determined based on constraints using a multi-objective optimization model. Based on the server load obtained from the monitoring system, servers that meet the requirements are selected, and greedy algorithms and genetic algorithms are used to optimize the solution and generate a deployment plan. The deployment schemes are scored using a fitness function, and the optimal scheme is selected and then deployed by encapsulating the data mining model.

[0017] According to a second aspect of the present invention, a unified encapsulation system for data mining algorithms is provided.

[0018] In one embodiment, the unified encapsulation system for the data mining algorithm includes: The algorithm encapsulation module is used to collect at least one data mining algorithm, analyze the data mining algorithm, construct a general interface template for the data mining algorithm based on the parameters obtained after analysis, encapsulate the data mining algorithm into algorithm components that match the general interface template, and build an evaluation model based on the algorithm components. The data mining evaluation module is used to perform calculations on all algorithm components, obtain component metadata files, and store the algorithm components and their metadata files in the target area. Through the graphical interface, the module uses the algorithm components to build pipelines, defines the execution order and dependencies of the algorithm components, obtains the data mining model, evaluates the model, and outputs an evaluation report of the data mining model. The data model optimization module is used to automatically adjust the data mining model based on a pre-configured data mining model through machine learning and optimization algorithms, and to call the evaluation model to evaluate the adjusted data mining model. The automatic deployment module uploads at least one tuned data mining model to the model repository, inputs the parameters of the data mining model, encapsulates the data mining model and deploys it to the target area, and analyzes the deployment strategy and automated deployment process of the data mining model in the target area.

[0019] According to a third aspect of the present invention, a computer device is provided.

[0020] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0021] According to a fourth aspect of the present invention, a computer-readable storage medium is provided.

[0022] In one embodiment, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the above method.

[0023] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: (1) By studying the unified programming paradigm of data mining, a unified interface that can cover different algorithms is extracted, and various mining algorithms are implemented based on this interface, so that users can call different algorithms with a unified interface. By wrapping the algorithm components with RESTful, the difference between the algorithm and the programming language of the caller is isolated. Calling in a service manner makes the algorithm components more applicable and facilitates the maintenance and expansion of data mining software.

[0024] (2) Lowering the barrier to entry for data mining, improving the interactive operation and user experience of machine learning, and increasing the efficiency of hybrid orchestration of feature processing and machine learning. This greatly improves the flexibility and maintainability of data mining.

[0025] (3) Reduce reliance on business and domain knowledge, lower the technical threshold, save time and resources, and improve model performance.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0028] Figure 1 This is a flowchart illustrating a unified encapsulation method for a data mining algorithm according to an exemplary embodiment; Figure 2 This is a principle block diagram of a unified encapsulation system for a data mining algorithm, illustrated according to an exemplary embodiment. Figure 3 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment; Figure 4 This is a schematic diagram of an architecture according to an exemplary embodiment. Detailed Implementation

[0029] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein encompasses the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0030] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings. They are used solely for the convenience of describing the document and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements; they can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0031] In this document, unless otherwise stated, the term "multiple" means two or more.

[0032] In this article, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0033] In this article, the term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0034] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0035] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0036] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0037] Figure 1An embodiment of a unified encapsulation method for a data mining algorithm according to the present invention is shown.

[0038] In this optional embodiment, the unified encapsulation method for a data mining algorithm includes: Step S101: Collect at least one data mining algorithm, analyze the data mining algorithm, construct a general interface template for the data mining algorithm based on the parameters obtained after analysis, encapsulate the data mining algorithm into algorithm components that match the general interface template, and construct an evaluation model based on the algorithm components. In this optional embodiment, the analysis of the input, output, and dependencies of the data mining algorithm identifies common requirements and differences among the data mining algorithms; the analysis of the input, output, and dependencies of the data mining algorithm identifies common requirements and differences among the data mining algorithms; the construction of basic and extended interfaces unifies the input and output of the interfaces, and generates algorithm metadata files for the data mining algorithm; the refactoring of the data mining algorithm code to match the interfaces, and the isolation of the data mining algorithm's runtime environment using virtual environments or containerization technology to obtain algorithm components; and the construction of an evaluation model based on pre-configured accuracy, computational efficiency, K-fold cross-validation, independent test set validation, learning curve analysis, regularization, and early stopping.

[0039] Specifically, we collect technical documentation and implementation code for data mining algorithms such as classification, clustering, regression, and association rules. The inputs for the analysis are: data format, feature columns, and hyperparameters, etc. The output includes: model files, prediction results, and evaluation metrics, etc. Dependencies include: data preprocessing requirements, etc. Common requirements include: data loading, model training, and prediction interface; The differences are in parameter types and calculation logic, etc.

[0040] Specifically, the basic interface includes essential methods such as fit() for training, predict() for prediction, and evaluate() for evaluation. The extended interface is for specific algorithms such as get_clusters() for clustering algorithms, generate_rules() for association rules, structured data objects such as DataFrame, and parameter configuration files such as JSON / YAML. The results object includes models, metrics, and visualization data, etc. Specifically, the steps for encapsulating the algorithm components are as follows: (1) Modify the code of each data mining algorithm to force the implementation of the fit, predict, and evaluate methods. For example, encapsulate Scikit-learn's SVM into the class SVMComponent.

[0041] (2) Dynamic loading of parameters: Hyperparameters are read from the configuration file through reflection mechanism to avoid hard coding.

[0042] (3) Dependency isolation: Use virtual environments (such as Python's venv) or containerization technologies (such as Docker) to isolate the running environment of different data mining algorithms and avoid dependency conflicts.

[0043] (4) Metadata generation: Generate metadata files for each data mining algorithm to describe its function, parameter range, and examples. For example, meta.json contains {"algorithm":"K-Means","params":{"n_clusters": "int"}}.

[0044] (5) RESTful service encapsulation: Each algorithm component is encapsulated as an HTTP service using a Web framework (such as FastAPI); Among them: the training interface: POST / algorithm / {name} / fit, which takes in the data and configuration and returns the model ID; the prediction interface: POST / algorithm / {name} / predict, which takes in the model ID and data and returns the prediction result; Input / output serialization: Data is transmitted in a standard format, supporting multilingual clients.

[0045] Service registration and discovery: Create a service registry (such as Consul or Nacos) to record the service address and metadata (such as description, input format and service address) of each data mining algorithm for dynamic querying by callers.

[0046] Specifically, the implementation of dynamic invocation and user interaction includes: (1) Create a unified API gateway to route all algorithm requests, for example, using Python. #Example of user call response = requests.post( "http: / / gateway / algorithm / kmeans / fit", json={"data":"data.csv","config": {"n_clusters": 3}} ) (2) Parameter dynamic parsing: The gateway obtains metadata from the service registry based on the algorithm name and verifies the legality of the parameters submitted by the user.

[0047] (3) Dynamic loading: Based on the algorithm type requested by the user, obtain the corresponding service endpoint from the registry center and dynamically load the parameter template.

[0048] (4) Self-explanatory function: The describe() method returns a link to the algorithm documentation, parameter descriptions and sample code to help users get started quickly. By writing developer guides, the interface specifications, component development process and debugging methods are explained in detail. By providing sample templates, code examples of typical scenarios are published. Through visualization tools, interactive documentation is integrated into the platform to allow users to debug the interface in real time.

[0049] (5) Standardized error handling: Standardize error codes and messages, such as: 400 Bad Request indicates parameter error, and 500 Internal Error indicates algorithm execution failure.

[0050] Specifically, user-visualized operations and automated script generation include: (1) Develop a web interface where users can select an algorithm via a drop-down menu, fill in parameters in a form, and trigger the call with a single click, for example: [Algorithm Selection: K-Means] [Parameter input: n_clusters=3] [Uploaded data: data.csv] [Click "Run"].

[0051] (2) Automatically generate calling code based on user operations for direct reuse by users, such as Python scripts: Python from sdk import AlgorithmClient client = AlgorithmClient() model_id = client.fit("kmeans", data="data.csv", config={"n_clusters": 3}) result = client.predict(model_id, test_data="test.csv").

[0052] Specifically, testing the stability, compatibility, and performance of the unified architecture includes: (1) Unit testing: Test the interface methods of each component, such as verifying whether fit() can correctly handle different data formats.

[0053] (2) Compatibility test: Verify whether different algorithms (such as SVM, random forest, etc.) can be successfully called through the same interface and whether the output format meets expectations.

[0054] (3) Integration testing: Simulate multi-component collaborative scenarios (such as classification model + feature engineering components) to verify the correctness of data flow and parameter transmission.

[0055] (4) Stress testing: Simulate high-concurrency scenarios (such as 1000 fit requests per second) and optimize service performance (such as introducing caching or asynchronous task queues).

[0056] (5) Performance optimization: Compare the execution efficiency of the algorithm before and after encapsulation, and optimize the serialization / deserialization process (such as using Protobuf instead of JSON).

[0057] (6) User acceptance testing: Invite target users to try out the interface, collect feedback and optimize the interaction design (such as simplifying the parameter configuration process).

[0058] Specifically, the evaluation of the model includes: model performance evaluation through accuracy and computational efficiency; generalization ability evaluation can be carried out by K-fold cross-validation, independent test set validation, learning curve analysis, and regularization and early stopping; accuracy evaluation can be carried out by selecting accuracy, precision, F1 score, ROC curve and AUC value and regression task; computational efficiency is achieved by recording training time, inference time and monitoring resource consumption, combined with data collected by monitoring tools.

[0059] (1) Performance evaluation formulas include: ; ; ; In the formula, Accuracy represents precision, Precision represents accuracy, Recall represents recall, F1 represents the harmonic mean of precision and recall, TP (True Positive) represents the number of samples correctly predicted as positive, TN (True Negative) represents the number of samples correctly predicted as negative, FP (False Positive) represents the number of samples incorrectly predicted as positive but actually negative, and FN (False Negative) represents the number of samples incorrectly predicted as negative but actually positive.

[0060] ROC curves and AUC values ​​refer to the area under the curve (AUC) calculated by plotting the true positive rate (TPR) and false positive rate (FPR) curves at different thresholds to evaluate overall performance.

[0061] The formula for the regression task is: ; In the formula, MSE represents the mean square error, and RMSE represents the root mean square error. N Indicates the number of samples.

[0062] R-squared formula: ; In the formula, R² represents the proportion of the variance of the explanatory variables to the target variable. This represents the true value of the i-th sample. This represents the predicted value of the i-th sample obtained through the regression model; Represents the true values ​​of all samples The mean.

[0063] Evaluation process: Input the test dataset, call the model's predict method to generate prediction results, select the indicator calculation module according to the task type, and output a standardized report.

[0064] Efficiency evaluation models include: 1) Record the time taken from calling the fit method to the completion of model training, record the average time taken for a single predict method, and monitor CPU / GPU utilization and memory usage.

[0065] 2) Embed timers during model training and prediction to record timestamps, and combine them with system monitoring tools (such as Prometheus) to collect resource usage data and generate efficiency reports (such as "training time: 120s, single prediction time: 0.5ms").

[0066] (2) The steps for the generalization ability assessment report include: 1) K-fold cross-validation: The dataset is randomly divided into k subsets of similar size. K-1 subsets are used as the training set in turn, and the remaining 1 subset is used as the validation set. The average of the k validation results is calculated as the final metric.

[0067] 2) Independent test set validation: The original dataset is divided into training set and test set according to the proportion. The test set is completely isolated during the training process. The model is trained only on the training set. Finally, the evaluate method is called on the test set to output the test set evaluation results.

[0068] 3) Learning curve analysis: By plotting the performance curves of the model on the training and validation sets as the amount of data increases, it is determined whether the model is overfitting or underfitting. The platform gradually increases the amount of training data and records the data, generates learning curve graphs based on the data, and provides analysis suggestions.

[0069] 4) Regularization and early stopping: Add weight penalty phase to loss function to suppress overfitting, monitor validation set loss, and terminate training early when there is no improvement after several consecutive rounds.

[0070] Evaluation process: Users can select the regularization type (such as L2) and strength through the configuration interface. During training, the validation set metrics are dynamically monitored, and training is terminated when the early stopping condition is triggered. The optimized model and training logs are then output.

[0071] Step S102: All algorithm components are computed to obtain component metadata files, and the algorithm components and their metadata files are stored in the target area. The algorithm components are used to build a pipeline through the graphical interface, the execution order and dependencies of the algorithm components are defined, the data mining model is obtained, and the evaluation report of the data mining model is output by evaluating the model. In this optional embodiment, the algorithm components are moved from the storage area to the graphical interface, and the algorithm components are connected according to the preset logic to form an algorithm component pipeline. The parameters of the algorithm components are set, the compatibility between the algorithm components is checked, and a data mining model is obtained. The pipeline is executed automatically, and the status of the algorithm components is monitored and recorded in real time.

[0072] Specifically, for each component's metadata file, the functionality, input / output format, and parameter configuration are described, for example: json { "name": "StandardScaler", "type": "feature_processing", "inputs": ["numeric_data"], "outputs": ["scaled_data"], "params": {"with_mean": true, "with_std": true} }; (1) Visual orchestration includes: moving the required algorithm components from the component library to the canvas to form an algorithm component pipeline, and connecting the components according to business logic, such as "data cleaning - feature engineering - training", and performing branching operations and loop logic according to condition judgment; The parameters of the algorithm component include: selecting feature columns and setting the number of clusters, and it supports dynamically passing parameters from the component, for example: yaml # Random Forest Parameter Configuration model_params: n_estimators: 100 max_depth: 10; (2) Allocate computing resources according to the type of algorithm components in the pipeline. For example, CPU-intensive feature processing uses multi-threading, GPU accelerates model training and supports distributed execution. At the same time, the output of each algorithm component serves as the input of downstream components, intermediate results are cached in the storage system, execution logs are recorded in real time, execution exceptions are captured and retry or skip options are provided when an exception occurs, and breakpoints are also supported.

[0073] (3) After the algorithm mining model is generated, the model evaluation report (such as the accuracy of the classification task and the ROC curve) and visualization chart (such as the ranking of feature importance) are output through the evaluation model. Users can download the results and adjust the process. At the same time, the platform records historical versions and supports comparison of performance differences.

[0074] Specifically, during the execution of the algorithm component, the system displays the component's execution status, resource usage, and time consumption in real time, and records detailed logs; it also supports pausing the process to check intermediate data.

[0075] Step S103: Based on the pre-configured data mining model, automatically adjust the data mining model through machine learning and optimization algorithms, and call the evaluation model to evaluate the adjusted data mining model; In this optional embodiment, the raw data is preprocessed, encapsulated into components using a general interface template, and the component features are described using metadata; derived features are automatically constructed based on domain knowledge, key features are selected, and candidate data mining models are trained according to the data scale and the complexity of key features; the hyperparameter range of the data mining algorithm model is defined, and the data mining algorithm model is adjusted using an optimization algorithm.

[0076] Specifically, the data preprocessing steps include: automatically identifying the data source and data type, detecting field types, automatically selecting strategies based on the proportion of missing data (such as deleting fields with high missing rates or filling in the mean, median, mode, and prediction models), using statistical methods or the isolated forest algorithm to identify outliers, automatically replacing or deleting outliers, and automatically selecting standardization methods based on the data mining model.

[0077] The derived features include: timestamp decomposition into year / month / day, text TF-IDF vectorization, feature cross or polynomial expansion, key features selection through statistical methods or model importance, recursive elimination of key features, automatic iterative selection of the optimal subset, integration of feature engineering library, dynamic orchestration of feature processing pipeline, and automatic combination of feature processing steps based on data characteristics using a dynamic orchestration engine.

[0078] Automated model selection includes: defining tasks based on the type of target variable or business needs, pre-setting commonly used algorithms (such as classification tasks: random forest, XGBoost, SVM; regression tasks: linear regression and GBRT, etc.), and quickly training candidate models based on data scale and feature complexity, before entering the adjustment phase.

[0079] Data mining model tuning includes: defining hyperparameter ranges for each model, selecting them through optimization algorithms (such as Bayesian optimization, genetic algorithms, and grid / random search algorithms), accelerating parallel training of multiple sets of parameters using a distributed computing framework, integrating them to obtain an optimization library, automatically recording the optimal parameter combination, allocating computing resources as needed using dynamic resource scheduling, and finally using an evaluation model to evaluate the tuned data mining model, generating an evaluation report and automatically comparing model performance, and outputting visual charts.

[0080] Specifically, during the execution of the algorithm component, the system displays the component's execution status, resource usage, and time consumption in real time, and records detailed logs; it also supports pausing the process to check intermediate data.

[0081] Step S104: Upload at least one adjusted data mining model to the model repository, input the parameters of the data mining model, encapsulate the data mining model and deploy it to the target area, and analyze the deployment strategy and automated deployment process of the data mining model in the target area.

[0082] In this optional embodiment, a pre-trained data mining model file is uploaded, and the file format and integrity are verified. Data mining model parameters are input and a version number is automatically generated. Based on a pre-selected deployment region, the data mining model is encapsulated. A mirror file of the data mining model is built using CI / CD tools and pushed. Based on the built mirror file, deployment is performed via Kubernetes. Based on the characteristics of the data mining model and the server status, the optimal solution and resource allocation scheme for server deployment of the data mining model are calculated, and automatic and flexible deployment is performed. Based on constraints, the optimization objective is determined through a multi-objective optimization model. Based on the server load obtained from the monitoring system, servers that meet the requirements are selected, and optimization solutions are obtained using greedy algorithms and genetic algorithms to generate deployment schemes. The deployment schemes are scored using a fitness function, and after selecting the optimal scheme, the data mining model is encapsulated and deployed.

[0083] Specifically, the parameters input to the data mining model include: hyperparameters, feature list, and input / output format.

[0084] After uploading a new model, the system automatically compares the differences between the old and new versions, updates the model's metadata, generates a new version number, retains the version rollback function, and verifies the compatibility of the new model with upstream and downstream components.

[0085] When managing models, the management interface displays basic model information, training time, author, performance metrics, and allows for dynamic parameter modification. It supports modifying runtime parameters via forms or APIs, and parameter changes are automatically synchronized to the deployment instance. It also supports metadata export, allowing model metadata to be exported as JSON / YAML files for integration with external systems.

[0086] Specifically, model deployment includes: (1) Select the target deployment server or cluster, configure resource quotas and the number of replicas, and encapsulate the model, for example: json { "data": [[1.2, 3.4, 5.6], [7.8, 9.0, 2.1]], "params": {"threshold": 0.7} }; The CI / CD tools are used to build a Docker image of the packaged model and push it to the repository. Kubernetes is used for deployment. After deployment, service availability is automatically detected, and real-time QPS, latency, and error rate are monitored. (2) Model monitoring and maintenance includes: real-time display of service request volume, response time and error logs, monitoring of model performance degradation, configuring alarm rules, automatically triggering rollback and cleaning up expired model versions in case of anomalies, releasing storage resources and updating model dependency libraries.

[0087] Specifically, analyzing optimal deployment strategies and automating the deployment process also includes: (1) Based on the model’s computational requirements, latency sensitivity, and priority, and the server status, such as hardware configuration, current load, cost, and geographical location, the optimal deployment server list and resource allocation scheme for each model are output.

[0088] (2) Based on the constraints: 1) server resource capacity limit, 2) redundancy deployment requirements of high-priority models, and 3) latency limit of real-time models, the objective function is used for optimization, and the formula is: ; In the formula, Cost represents the total deployment cost, Latency represents the weighted average of model inference latency, Resource_Waste represents the penalty for unused resources, and α, β, and γ represent weight coefficients that are adjusted according to business needs.

[0089] Specifically, real-time server load is obtained from a monitoring system (such as Prometheus), the model configuration file is parsed, resource requirements and business rules are extracted, servers that do not meet the minimum hardware requirements are excluded, and the remaining servers are scored according to dimensions such as resource availability, cost, and latency.

[0090] A greedy algorithm is used to prioritize the models, select the current best server in turn, update the server resource reserves, and iterate the allocation. A genetic algorithm is used to represent the deployment scheme as chromosomes, and a fitness function is used to calculate the score of the scheme. Finally, new schemes are generated through crossover and mutation, and the process is iteratively evolved.

[0091] The mapping relationship between the output model and the server, for example: json { "model_id": "fraud_detection_v1", "primary_server": "gpu-node-01", "backup_servers": ["gpu-node-02"], "resource_allocation": {"gpu": 1, "memory": "8Gi"} }; After encapsulating the model, deploy it via Kubernetes, for example: yaml apiVersion: apps / v1 kind: Deployment metadata: name: fraud-detection spec: replicas: 2 template: spec containers: - name: model-server image: registry / fraud_detection:v1 resources: limits nvidia.com / gpu: 1 memory: 8Gi The automated triggering conditions are: when the algorithm generates a new deployment strategy and when server failure or resource overrun is detected; when the triggering conditions are detected, the Kubernetes API is called to create / update the Deployment, verify the service health status, and update the model manager metadata.

[0092] Figure 2 This invention illustrates an embodiment of a unified encapsulation system for a data mining algorithm according to the present invention.

[0093] In this optional implementation, the unified encapsulation system for a data mining algorithm includes: The algorithm encapsulation module 201 is used to collect at least one data mining algorithm, analyze the data mining algorithm, construct a general interface template for the data mining algorithm based on the parameters obtained after analysis, encapsulate the data mining algorithm into an algorithm component that matches the general interface template, and construct an evaluation model based on the algorithm component. The data mining evaluation module 202 is used to perform calculations on all algorithm components, obtain component metadata files, and store the algorithm components and their metadata files in the target area. Through the graphical interface, the algorithm components are used to build pipelines, define the execution order and dependencies of the algorithm components, obtain the data mining model, and output the evaluation report of the data mining model through the evaluation model. The data model optimization module 203 is used to automatically adjust the data mining model based on a pre-configured data mining model through machine learning and optimization algorithms, and to call the evaluation model to evaluate the adjusted data mining model. The automatic deployment module 204 uploads at least one adjusted data mining model to the model repository, inputs the parameters of the data mining model, encapsulates the data mining model and deploys it to the target area, and analyzes the deployment strategy and automated deployment process of the data mining model in the target area.

[0094] Figure 4 This invention illustrates the technical architecture design of a unified encapsulation system for a data mining algorithm according to the present invention.

[0095] In this example, dashed boxes represent third-party applications, while solid boxes represent applications provided by this platform. The application layer provides browser-based user interaction operations, including data mining applications and data mining integrated development environments, the latter of which can run independently or be invoked by the former via a URL.

[0096] The control layer provides design, orchestration, automation, and model management capabilities on the backend, including components such as a data mining designer, a data mining auto-orchestrator, and a data mining executor. The data mining executor is a runtime component, while the others are design-time components. The data mining auto-orchestrator automatically orchestrates data based on the purpose and characteristics of the data analysis, using templates and self-learning algorithms. It combines feature processing with one or more required machine learning algorithms and automatically sets hyperparameters.

[0097] The model layer provides components related to feature processing, mining algorithms, and mining evaluation. These include feature processing and mining algorithm components based on a unified interface, as well as a mining evaluator that provides cross-evaluation capabilities in addition to assessing data mining performance. The feature processing and mining algorithm components will provide interfaces externally as RESTful services. The model layer also provides a model manager, offering functions such as adding, updating, deleting, and searching models.

[0098] The data resource layer provides a dataset provider that prepares datasets for model training through a unified interface. The dataset provider introduces external data access and data integration components, providing data support for data analysis and mining.

[0099] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores static and dynamic information data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0100] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0101] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0102] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0104] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A unified encapsulation method for data mining algorithms, characterized in that, include: Collect at least one data mining algorithm, analyze the data mining algorithm, construct a general interface template for the data mining algorithm based on the parameters obtained after analysis, encapsulate the data mining algorithm into algorithm components that match the general interface template, and build an evaluation model based on the algorithm components. Calculate all algorithm components to obtain component metadata files, and store the algorithm components and their metadata files in the target area. Use the algorithm components to build a pipeline through a graphical interface, define the execution order and dependencies of the algorithm components, obtain the data mining model, evaluate the model, and output an evaluation report of the data mining model. Based on a pre-configured data mining model, the data mining model is automatically adjusted through machine learning and optimization algorithms, and an evaluation model is invoked to evaluate the adjusted data mining model. Upload at least one well-tuned data mining model to the model repository, input the parameters of the data mining model, package and deploy the data mining model to the target region, and analyze the deployment strategy and automated deployment process of the data mining model in the target region.

2. The unified encapsulation method for data mining algorithms according to claim 1, characterized in that, The process of collecting at least one data mining algorithm, analyzing the data mining algorithm, constructing a general interface template for the data mining algorithm based on the parameters obtained from the analysis, encapsulating the data mining algorithm into algorithm components that match the general interface template, and constructing an evaluation model based on the algorithm components includes: Analyze the inputs, outputs, and dependencies of data mining algorithms to identify common requirements and differences among them. Build basic and extended interfaces, unify the input and output of the interfaces, and generate algorithm metadata files for data mining algorithms; The data mining algorithm is refactored to match its interface, and the running environment of the data mining algorithm is isolated using virtual environment or containerization technology to obtain the algorithm component.

3. The unified encapsulation method for data mining algorithms according to claim 2, characterized in that... The construction of basic and extended interfaces, the unification of interface input and output, and the generation of algorithm metadata files for data mining algorithms include: The basic interface includes preset necessary processing methods, while the extended interface is set based on the data mining algorithm type; The interface input is uniformly structured data objects and parameter configuration files, and the interface output is uniformly result objects.

4. The unified encapsulation method for data mining algorithms according to claim 2, characterized in that... The evaluation model built based on algorithm components includes: The evaluation model is built based on pre-configured accuracy, computational efficiency, K-fold cross-validation, independent test set validation, learning curve analysis, regularization, and early stopping.

5. The unified encapsulation method for data mining algorithms according to claim 4, characterized in that, The evaluation model, based on pre-configured accuracy, computational efficiency, K-fold cross-validation, independent test set validation, learning curve analysis, and regularization and early stopping, includes: Accuracy is assessed based on accuracy, precision, F1 score, ROC curve and AUC value, and regression task. Data is collected by recording training time, inference time, monitoring resource consumption and computational efficiency, and combining this with monitoring tools.

6. The unified encapsulation method for data mining algorithms according to claim 1, characterized in that, The process involves calculating all algorithm components to obtain component metadata files, storing the algorithm components and their metadata files in the target area, constructing a pipeline using the algorithm components through a graphical interface, defining the execution order and dependencies of the algorithm components, obtaining a data mining model, and evaluating the model to output an evaluation report, including: The algorithm components are moved from the storage area to the graphical interface, and the algorithm components are connected according to the preset logic to form an algorithm component pipeline. The parameters of the algorithm components are set, the compatibility between the algorithm components is checked, and the data mining model is obtained. The pipeline is executed automatically, and the status of the algorithm components is monitored and recorded in real time.

7. The unified encapsulation method for data mining algorithms according to claim 1, characterized in that, The process of automatically adjusting the pre-configured data mining model using machine learning and optimization algorithms, and then evaluating the adjusted model using an evaluation model, includes: The raw data is preprocessed, encapsulated into components using a common interface template, and the component characteristics are described using metadata. Based on domain knowledge, we automatically construct derived features, filter out key features, and train candidate data mining models according to the data scale and the complexity of key features. Define the hyperparameter range of the data mining algorithm model, and use optimization algorithms to adjust the data mining algorithm model.

8. The unified encapsulation method for data mining algorithms according to claim 1, characterized in that, The process of uploading at least one tuned data mining model to the model repository, inputting the parameters of the data mining model, packaging and deploying the data mining model to the target region, and analyzing the deployment strategy and automated deployment process of the data mining model in the target region includes: Upload the pre-trained data mining model file, verify the format and integrity of the data mining model file, input the data mining model parameters, and automatically generate the version number; Based on a pre-selected deployment region, the data mining model is packaged, and an image file of the data mining model is built by calling CI / CD tools and pushed. Based on the built image file, it is deployed via Kubernetes. Based on the characteristics of the data mining model and the server status, the optimal solution and resource allocation scheme for the data mining model on the server are calculated, and automatic and flexible deployment is performed.

9. The unified encapsulation method for data mining algorithms according to claim 8, characterized in that, The process of calculating the optimal solution for server deployment and resource allocation scheme based on the characteristics of the data mining model and the server status, and then automatically and flexibly deploying it, includes: The optimization objective is determined based on constraints using a multi-objective optimization model. Based on the server load obtained from the monitoring system, servers that meet the requirements are selected, and greedy algorithms and genetic algorithms are used to optimize the solution and generate a deployment plan. The deployment schemes are scored using a fitness function, and the optimal scheme is selected and then deployed by encapsulating the data mining model.

10. A unified encapsulation system for data mining algorithms, characterized in that, The system includes: The algorithm encapsulation module is used to collect at least one data mining algorithm, analyze the data mining algorithm, construct a general interface template for the data mining algorithm based on the parameters obtained after analysis, encapsulate the data mining algorithm into algorithm components that match the general interface template, and build an evaluation model based on the algorithm components. The data mining evaluation module is used to perform calculations on all algorithm components, obtain component metadata files, and store the algorithm components and their metadata files in the target area. Through the graphical interface, the module uses the algorithm components to build pipelines, defines the execution order and dependencies of the algorithm components, obtains the data mining model, evaluates the model, and outputs an evaluation report of the data mining model. The data model optimization module is used to automatically adjust the data mining model based on a pre-configured data mining model through machine learning and optimization algorithms, and to call the evaluation model to evaluate the adjusted data mining model. The automatic deployment module uploads at least one tuned data mining model to the model repository, inputs the parameters of the data mining model, encapsulates the data mining model and deploys it to the target area, and analyzes the deployment strategy and automated deployment process of the data mining model in the target area.

Images