Algorithm configuration combination platform based on data lake server

Abstract

The present invention is an algorithm configuration combination platform based on a data lake server, comprising: a data directory area, a data preprocessing workbench, a data preprocessing resource directory area, a general algorithm operation directory area, an algorithm flow workbench, a model resource directory area, and a knowledge base; its operation method comprises: step 1, at a data lake server node, entering the algorithm configuration combination platform; step 2, saving the data directory to the data directory area, and selecting a number of data to move to the data preprocessing workbench for preprocessing, and saving them to the data preprocessing resource directory area; step 3, judging whether to use the training model generated in the model resource directory area, if so, executing step 4, if not, executing a custom algorithm operation process; step 4, selecting a training model in the model resource directory area, and selecting unclassified data in the data preprocessing resource directory area; step 5, executing the training model, and saving the generated calculation results to the knowledge base.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an algorithm configuration combination platform based on a data lake server. Background Art

[0002] Database technology is the foundation and core of modern computer information systems and computer application systems, and is an important part of information systems. When developing database application systems, it is usually necessary to export database data for system backup or data sharing and exchange with other systems.

[0003] The concept of data lake or hub was first proposed by big data vendors. On the surface, data is carried on cheap storage hardware based on scalable HDFS (Hadoop Distributed File System). But the larger the amount of data, the more different types of storage are needed. Ultimately, all enterprise data can be considered big data, but not all enterprise data is suitable for storage on cheap HDFS clusters. Part of the value of data lake is to bring different types of data together, and another part of the value is that data analysis can be performed without predefined models. Today's big data architecture is scalable and can provide users with more and more real-time analysis. The data lake architecture is aimed at information storage from multiple data sources, including the Internet of Things. Big data analysis or archiving can be processed or delivered to requesting users by accessing the data lake.

[0004] At present, the design of algorithms related to data lakes relies solely on experience and requires repeated testing and modification. The complexity of the algorithm will significantly affect its generalization ability, consume a lot of working time, and finally make it lack of reusability. Therefore, it is of great significance to simplify algorithm design and reuse existing algorithms. Summary of the invention

[0005] The present invention proposes an algorithm configuration combination platform based on a data lake server, so that the algorithms designed on the platform can effectively shorten the time and simplify the design of the algorithms.

[0006] In order to achieve the above objectives, the present invention is implemented by the following technical solutions:

[0007] An algorithm configuration combination platform based on a data lake server, which features include: a data directory area, a data preprocessing workbench, a data preprocessing resource directory area, a general algorithm operation directory area, an algorithm flow workbench, a model resource directory area, and a knowledge base;

[0008] The data directory area stores the data directory created when the user uploads data;

[0009] The data preprocessing workbench will read the data stored in several data directories, and form data preprocessing resources after preprocessing, and save the generated data preprocessing resources to the data preprocessing resource directory area;

[0010] The general algorithm operation directory area stores a number of algorithm operations and transmits one or more algorithm operations to the algorithm flow workbench;

[0011] Select the order of algorithm operations in the algorithm process workbench and input the threshold required by the algorithm;

[0012] Select a data preprocessing resource from the data preprocessing resource directory area as the training set data for the first algorithm operation, and start the training algorithm process;

[0013] If the algorithm flow is an algorithm flow of a calculation result, the calculation result is saved in the knowledge base;

[0014] If the algorithm flow is an algorithm flow of a training model, the training model is saved to the model resource directory area;

[0015] The generated training model is selected from the model resource directory area, and the unclassified data in the data preprocessing resource directory area is selected for calculation, and the generated calculation results are saved in the knowledge base. The generated model and data related to the model are stored in the distributed file system in the data lake server in the form of files, and will be used as the model source and data source of the data visualization editing and display platform when needed in the future.

[0016] Preferably, the algorithm flow for calculating the result includes clustering and association rule algorithms.

[0017] The algorithm flow of training the model includes classification, regression, artificial neural network algorithm and data feature transformation operations.

[0018] Preferably, the data feature transformation operations provided in the general algorithm operation catalog area further include: dimensionality reduction, binning, and normalization.

[0019] Preferably, the data preprocessing includes data screening and grouping evaluation based on threshold values;

[0020] If no threshold is entered in the algorithm flow workbench, a default value will be used instead.

[0021] Preferably, the data stored in the data directory includes tables, fields and data tags.

[0022] Preferably, the data owner logs in to the graphical operation interface with a user name and password on a data lake server node, accesses the data lake server management platform, and then clicks a button to configure the algorithm combination platform;

[0023] The data directory area, data preprocessing workbench, data preprocessing resource directory area, general algorithm operation directory area, algorithm flow workbench, model resource directory area, knowledge base, and console are all graphical interfaces.

[0024] Preferably, a console is also included, and the results or abnormal results and error messages that appear during the execution of the training algorithm process are printed to the console and saved as a log.

[0025] Preferably, the data lake server includes a data storage and management service platform consisting of a relational database, a document database, a distributed file system, and a graph database, and provides multiple functional components including data management and algorithm development;

[0026] The management service platform adopts a distributed computing and storage architecture, integrating a variety of stand-alone computers, servers, and computer clusters or server clusters with data storage and computing functions.

[0027] A method for operating an algorithm configuration combination platform based on a data lake server, wherein steps of a custom algorithm operation process in the algorithm configuration combination platform include:

[0028] Step 3.1, the user selects several algorithm operations that need to be combined from the general algorithm operation catalog area and transfers them to the algorithm flow workbench;

[0029] Step 3.2: Select the order of algorithm operations in the algorithm process workbench and input the threshold required by the algorithm; if the threshold is not input, the default value is used to form the algorithm operation process;

[0030] Step 3.3: The user selects a training data source applied to the algorithm operation process from the data preprocessing resource directory area to perform model training and obtain results or train the model;

[0031] Step 3.4: If the user uses classification, regression, neural network algorithms and data feature transformation operations, he can upload the data to be predicted or classified, and specify the training model of the above algorithm operation for prediction or classification.

[0032] Preferably, in step 3.3, if the algorithm flow is an algorithm flow for calculating results, the results are directly generated and saved to the algorithm configuration combination platform knowledge base; if the algorithm flow is an algorithm flow for training models, a model is generated and saved to the model resource directory area.

[0033] Preferably, the operation process of the algorithm configuration combination platform includes:

[0034] Step 1: The data owner logs in to the graphical operation interface of a data lake server node with a user name and password, accesses the data lake server management platform, and then enters the algorithm configuration combination platform;

[0035] Step 2: The data owner saves the data directory created when uploading the data to the data directory area, selects one or more data and moves them to the data preprocessing workbench for preprocessing to form data preprocessing resources, and saves the generated data preprocessing resources to the data preprocessing resource directory area;

[0036] Step 3: Determine whether to use the training model generated in the model resource directory area. If yes, execute step 4. If not, execute the custom algorithm operation process, including the appropriate modeling platform and software version required for the custom algorithm process.

[0037] Step 4: Select an existing training model in the model resource directory area, and select unclassified data in the data preprocessing resource directory area;

[0038] Step 5: Execute the training model and save the generated calculation results to the knowledge base.

[0039] The present invention has the advantages of simplifying the design of the algorithm and providing a reusable algorithm design platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a diagram of the data lake architecture;

[0041] Figure 2 It is an operation flow chart of the present invention. DETAILED DESCRIPTION

[0042] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments, but the scope of the present invention is not limited in any way.

[0043] As attached Figure 1 As shown, the data lake server includes a data storage and management service platform consisting of a relational database, a document database, a distributed file system, and a graph database, and provides multiple functional components including data management and algorithm development; the relational database includes MariaDB and MySQL;

[0044] The document-type database includes MongoDB and CouchDB; the distributed file system includes HDFS, PVFS, and PanFS; the graph database includes Neo4j, Cayley, and GrapgDB; the management service platform adopts a distributed computing and storage architecture, integrating a variety of stand-alone computers, servers, and computer clusters or server clusters with data storage and computing functions.

[0045] Data exchange management of the entire data lake is based on log records and file metadata stored in MongoDB.

[0046] 1. Log record data exists in the form of key-value pairs, and its field names and contents are:

[0047] Field "user": saves the name of the operator of the current operation;

[0048] Field "operation_type": saves the type of the current operation, such as create, modify, append, etc.;

[0049] Field "operation_record": used to save the content of the current operation, i.e. the object of the operation. When the operation type is modification, creation, or append, the location of the corresponding data source is saved; when the operation type is query, the corresponding query statement is saved;

[0050] Field "operation_time": saves the date and time of the current operation, for example: "2018-06-28T03:18:58.91;

[0051] Field "operation_statue": saves the status of the current operation. It is an auxiliary field used to determine whether the current operation is successful;

[0052] Field "operation_source": saves the data type of the current operation, for example: "hdfs" represents file-type data.

[0053] 2. File metadata exists in the form of key-value pairs, and its field names and contents are mainly:

[0054] Data name: the name of the data being processed;

[0055] Description: Description of the current data;

[0056] Owner: the user to whom the current data belongs;

[0057] Group: The group to which the current data belongs;

[0058] Storage backend: the storage destination, which refers to a certain database type;

[0059] Auxiliary tags: RDF generated by data;

[0060] Metadata creation time: the time when the metadata was created;

[0061] Metadata update time: the time when the metadata is updated.

[0062] The above-mentioned "storage backend" field includes different fields according to different data types (file type, document type, table type, and graph type). For file-type data, there are the following fields: file physical path, file physical name, HDFS occupied space size, real file owner, real file group, front-end displayed file path, front-end displayed file name, file extension, file MINE type, file real size, stop word list (for RDF processing); for document-type data (JSON-type data), there are the following fields: physical database location, physical collection name, display database name, display collection name, document structure (JSON data structure), stop word list (for RDF processing); for table-type data (such as data from mySql), there are the following fields: physical database name, physical table name, display database name, display table name, column name list, stop word list (for RDF processing); for graph-type data (such as data from Neo4j), there are the following fields: Neo4j ID, front-end displayed name, stop word list (for RDF processing); through the above-mentioned log files and metadata files, the data management service platform can efficiently and securely organize and manage data files and speed up data storage.

[0063] An algorithm configuration combination platform based on a data lake server, which features: data directory area, data preprocessing workbench, data preprocessing resource directory area, general algorithm operation directory area, algorithm flow workbench, model resource directory area, console, knowledge base. The data owner logs in to the graphical operation interface with a user name and password on a data lake server node, accesses the data lake server management platform, and then clicks the button algorithm configuration combination platform; the data directory area, data preprocessing workbench, data preprocessing resource directory area, general algorithm operation directory area, algorithm flow workbench, model resource directory area, knowledge base, and console are all graphical interfaces.

[0064] The data directory area stores the data directory created when the user uploads data; the data preprocessing workbench reads the data stored in several data directories, forms data preprocessing resources after preprocessing, and stores the generated data preprocessing resources in the data preprocessing resource directory area; the data stored in the data directory includes tables, fields and data labels; the data preprocessing work includes data screening and group evaluation based on thresholds.

[0065] The general algorithm operation directory area stores several algorithm operations and passes one or more algorithm operations to the algorithm flow workbench; the order of algorithm operations is selected in the algorithm flow workbench, and the threshold required by the algorithm is input; if the threshold is not input, it is replaced by a default value; a data preprocessing resource is selected in the data preprocessing resource directory area as the training set data for the first algorithm operation, and the training algorithm flow is started; if the algorithm flow is an algorithm flow of clustering or association rule algorithm, its calculation result is saved to the knowledge base; the results or abnormal results and error information appearing in the process of executing the training algorithm flow are printed to the console and saved as a log; if the algorithm flow is an algorithm flow of classification, regression, artificial neural network algorithm and data feature transformation operation, the training model is saved to the model resource directory area; the generated training model is selected in the model resource directory area, and the unclassified data in the data preprocessing resource directory area is selected for calculation, and the generated calculation results are saved to the knowledge base; the saving to the knowledge base means that the generated model and the data related to the model are stored in the distributed file system in the data lake server in the form of files, and used as the model source and data source of the data visualization editing and display platform when needed in the future.

[0066] As attached Figure 2 As shown, the embodiment of the present invention is that the user uses the data lake server to configure the algorithm combination platform, and finally forms an algorithm model. The method includes the following steps:

[0067] Step 1: The data owner logs in to the graphical operation interface of a data lake server node with a user name and password, accesses the data lake server management platform, and then enters the algorithm configuration combination platform;

[0068] Step 2: The data owner saves the data directory created when uploading the data to the data directory area, selects one or more data and moves them to the data preprocessing workbench for preprocessing to form data preprocessing resources, and saves the generated data preprocessing resources to the data preprocessing resource directory area;

[0069] Step 3: Determine whether to use the training model generated in the model resource directory area. If yes, execute step 4. If not, execute the custom algorithm operation process, including the appropriate modeling platform and software version required for the custom algorithm process.

[0070] Step 4: Select an existing training model in the model resource directory area, and select unclassified data in the data preprocessing resource directory area;

[0071] Step 5: Execute the training model and save the generated calculation results to the knowledge base.

[0072] The custom algorithm operation process includes the following steps:

[0073] Step 3.1, the user selects several algorithm operations that need to be combined from the general algorithm operation catalog area and passes them to the algorithm process workbench; the general algorithm operation catalog area provides users with several major types of algorithms such as classification, clustering, association rules, regression, and neural networks, as well as data feature transformation operations such as dimensionality reduction, binning, and normalization, collectively referred to as algorithm operations, and users can customize the serial combination and arrangement of the above operations;

[0074] Step 3.2: Select the order of algorithm operations in the algorithm process workbench and input the threshold required by the algorithm; if the threshold is not input, the default value is used to form the algorithm operation process;

[0075] Step 3.3, the user selects a training data source applied to the algorithm operation process from the data preprocessing resource directory area to perform model training, obtain results or train the model; if the algorithm process is an algorithm process for calculating results, the result is directly generated and saved to the algorithm configuration combination platform knowledge base; if the algorithm process is an algorithm process for training models, a model will be generated and saved to the model resource directory area;

[0076] Step 3.4: If the user uses classification, regression, neural network algorithms and data feature transformation operations, he can upload the data to be predicted or classified, and specify the training model of the above algorithm operation for prediction or classification.

[0077] The present invention has the advantages of simplifying the design of the algorithm and providing a reusable algorithm design platform.

[0078] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art for various modifications and substitutions of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. An algorithm configuration combination platform based on a data lake server, characterized in that: Contains: data directory area, data preprocessing workbench, data preprocessing resource directory area, general algorithm operation directory area, algorithm flow workbench, model resource directory area, knowledge base; The data directory area stores the data directory created when the user uploads data; The data preprocessing workbench will read the data stored in several data directories, and form data preprocessing resources after preprocessing, and save the generated data preprocessing resources to the data preprocessing resource directory area; The general algorithm operation directory area stores a number of algorithm operations and transmits one or more algorithm operations to the algorithm flow workbench; Select the order of algorithm operations in the algorithm process workbench and input the threshold required by the algorithm; Select a data preprocessing resource from the data preprocessing resource directory area as the training set data for the first algorithm operation, and start the training algorithm process; If the algorithm flow is an algorithm flow of a calculation result, the calculation result is saved in the knowledge base; If the algorithm flow is an algorithm flow of a training model, the training model is saved to the model resource directory area; The generated training model is selected from the model resource directory area, and the unclassified data in the data preprocessing resource directory area is selected for calculation, and the generated calculation results are saved in the knowledge base. The generated model and data related to the model are stored in the distributed file system in the data lake server in the form of files, and will be used as the model source and data source of the data visualization editing and display platform when needed in the future.

2. The algorithm configuration combination platform based on the data lake server as claimed in claim 1, characterized in that: The algorithm flow for calculating the results includes clustering and association rule algorithms; The algorithm flow of training the model includes classification, regression, artificial neural network algorithm and data feature transformation operations.

3. The algorithm configuration combination platform based on the data lake server according to claim 2, characterized in that: The data feature transformation operations provided in the general algorithm operation directory area further include: dimensionality reduction, binning, and normalization.

4. The algorithm configuration combination platform based on the data lake server as described in claim 1, characterized in that: The data preprocessing work includes data screening and grouping evaluation by threshold; If no threshold is entered in the algorithm flow workbench, a default value will be used instead.

5. The algorithm configuration combination platform based on the data lake server as claimed in claim 1, characterized in that: The data stored in the data directory includes tables, fields and data tags.

6. The algorithm configuration combination platform based on the data lake server as claimed in claim 1, characterized in that: The data owner logs in to the graphical operation interface of a data lake server node with a username and password, accesses the data lake server management platform, and then clicks the button algorithm configuration combination platform; The data directory area, data preprocessing workbench, data preprocessing resource directory area, general algorithm operation directory area, algorithm flow workbench, model resource directory area, knowledge base, and console are all graphical interfaces.

7. The algorithm configuration combination platform based on the data lake server according to claim 1, characterized in that: It also includes a console, and the results or abnormal results and error messages that appear during the execution of the training algorithm process are printed to the console and saved as a log.

8. The algorithm configuration combination platform based on the data lake server as claimed in claim 1, characterized in that: The data lake server includes a data storage and management service platform consisting of a relational database, a document database, a distributed file system, and a graph database, and provides multiple functional components including data management and algorithm development; The management service platform adopts a distributed computing and storage architecture, integrating a variety of stand-alone computers, servers, and computer clusters or server clusters with data storage and computing functions.

9. A method for operating an algorithm configuration combination platform based on a data lake server, characterized in that: Using the algorithm configuration combination platform based on a data lake server as described in any one of claims 1 to 8, the steps of customizing the algorithm operation process in the algorithm configuration combination platform include: Step 3.1, the user selects several algorithm operations that need to be combined from the general algorithm operation catalog area and transfers them to the algorithm flow workbench; Step 3.2: Select the order of algorithm operations in the algorithm process workbench and input the threshold required by the algorithm; if the threshold is not input, the default value is used to form the algorithm operation process; Step 3.3: The user selects a training data source applied to the algorithm operation process from the data preprocessing resource directory area to perform model training and obtain results or train the model; Step 3.4: If the user uses classification, regression, neural network algorithms and data feature transformation operations, he can upload the data to be predicted or classified, and specify the training model of the above algorithm operation for prediction or classification.

10. The method for operating an algorithm configuration combination platform based on a data lake server according to claim 9, characterized in that: In step 3.3, if the algorithm flow is an algorithm flow for calculating results, the results are directly generated and saved to the algorithm configuration combination platform knowledge base; If the algorithm flow is the algorithm flow of training a model, a model will be generated and saved in the model resource directory area.

11. The method for operating an algorithm configuration combination platform based on a data lake server according to claim 10, characterized in that: The operation process of the algorithm configuration combination platform includes: Step 1: The data owner logs in to the graphical operation interface of a data lake server node with a user name and password, accesses the data lake server management platform, and then enters the algorithm configuration combination platform; Step 2: The data owner saves the data directory created when uploading the data to the data directory area, selects one or more data and moves them to the data preprocessing workbench for preprocessing to form data preprocessing resources, and saves the generated data preprocessing resources to the data preprocessing resource directory area; Step 3: Determine whether to use the training model generated in the model resource directory area. If yes, execute step 4. If not, execute the custom algorithm operation process, including the appropriate modeling platform and software version required for the custom algorithm process. Step 4: Select an existing training model in the model resource directory area, and select unclassified data in the data preprocessing resource directory area; Step 5: Execute the training model and save the generated calculation results to the knowledge base.

Images