Construction system and method for big data analysis algorithm library

Abstract

Disclosed in the present invention is a construction system for a big data analysis algorithm library, comprising: an analysis process construction module, an operator platform selection module, a code reverse generation module, a user code verification module, a cluster control host, and a cluster computing host. Cross-platform and cross-language algorithm fusion can be realized, and data analysis is realized in a complete and heterogeneous data analysis flow. Also disclosed in the present invention is a construction method for a big data analysis algorithm library.

Description

Construction system and construction method of big data analysis algorithm library Technical Field

[0001] The present invention belongs to the technical field of big data analysis methods, and relates to a system for constructing a big data analysis algorithm library. The present invention also relates to a method for constructing a big data analysis algorithm library. Background Art

[0002] With the development of big data ecological software, big data analysis algorithms based on big data technology and centered on data processing and business analysis have also been widely used. Therefore, building an algorithm library for industry-oriented big data analysis is one of the necessary elements to promote the development of the big data industry.

[0003] Traditional big data algorithm libraries are mostly designed to enhance the richness of platform algorithms, but they fall short of fully meeting these requirements in terms of cross-platform algorithm heterogeneity, multi-language compatibility, algorithm selection, and flexible algorithm usage. Common data analysis algorithms are all developed based on a unified language and environment. Examples include small-scale data analysis algorithm libraries like Scikit-learn, large-scale data analysis algorithm libraries like Spark MLlib, statistical analysis algorithm libraries like MATLAB, deep learning algorithm frameworks like Pytorch and TensorFlow, and signal data processing algorithm libraries like H2O. However, in real-world industry data analysis, multiple algorithms from multiple platforms often need to be integrated to build a complete data analysis process. This requires users to first select a platform based on their business needs and then choose algorithms based on those platforms. If multiple algorithms required by a process exist across multiple platforms, manual coding is required to ensure consistency across these environments. Furthermore, traditional enterprises have accumulated a large number of proprietary algorithms to solve industry-specific data analysis problems over the course of their development. These industry-specific algorithms may be based on any programming environment, requiring code refactoring to adapt them to the entire analysis process when integrated into an algorithm library. All of the above undoubtedly brings difficulties to the establishment of a big data analysis algorithm library for enterprises.

[0004] In summary, existing big data analysis algorithm libraries or big data analysis platforms have the following main problems in industry-oriented data analysis scenarios:

[0005] Analysis efficiency issues: Traditional data analysis requires cross-disciplinary experts to spend a lot of time on business communication and algorithm debugging. The threshold is high, which causes many inconveniences in analysis modeling and design.

[0006] Algorithm selection problem: Traditional data analysis cannot combine the advantages of the algorithms of various open source platforms and automatically select operators for specific platforms. A lot of manual intervention is required to select the appropriate platform and further select the appropriate algorithm.

[0007] Algorithm adaptation problem: Traditional data analysis platforms provide a fully configured interface, which prevents users from completing customized code modifications. The interface call format is difficult for novice users to use, and lacks flexibility in the data analysis process.

[0008] Algorithm fusion problem: Traditional analysis algorithm platforms cannot support the technical interoperability and seamless connection between new platform algorithms and the company's historical algorithms, and thus cannot achieve broad compatibility and sustainable evolution of the industry's own algorithm library. Summary of the Invention

[0009] The purpose of this invention is to provide a system for building a big data analysis algorithm library, which can realize cross-platform and cross-language algorithm fusion and implement data analysis in a complete and heterogeneous data analysis flow.

[0010] Another object of the present invention is to provide a method for constructing a big data analysis algorithm library.

[0011] The technical solution adopted by the present invention is a system for constructing a big data analysis algorithm library, including:

[0012] The analysis process building module allows users to visually select algorithms and connect them end to end to build a complete data analysis process and generate a process description file;

[0013] The operator platform selection module selects a benchmark operating environment for each algorithm in the data analysis process built by the analysis process construction module, and writes the benchmark operating environment corresponding to each algorithm into the process description file;

[0014] The code reverse generation module generates the algorithm code of each algorithm in its benchmark operating environment, which can be manually edited by users;

[0015] User code verification module verifies whether the algorithm code manually edited by the user complies with the rules. If it does, it generates native code;

[0016] The cluster control host is deployed with a process execution engine, which sends the native code of each algorithm to the cluster computing host that matches its environment;

[0017] The cluster computing host receives the native code issued by the process execution engine, completes data reading and input, code compilation and execution, and data output and caching.

[0018] The present invention is also characterized in that: the cluster computing hosts include multiple categories, each category of cluster computing hosts deploys only one benchmark operating environment, and the cluster control host selects the corresponding cluster computing host according to the benchmark operating environment of the algorithm to issue native code.

[0019] Another technical solution adopted by the present invention is a method for constructing a big data analysis algorithm library, which adopts the above-mentioned big data analysis algorithm library construction system and is implemented in the following steps:

[0020] Step 1: The algorithm library initially includes various algorithms used in big data analysis;

[0021] Step 2: Users visually edit data analysis process nodes through the analysis process building module according to their needs to form a data analysis process. Each process node corresponds to an algorithm in step 1, and a process description file is generated;

[0022] Step 3: The operator platform selection module selects a suitable benchmark operating environment for the algorithm of this process node and writes the benchmark operating environment corresponding to each algorithm into the process description file;

[0023] Step 4: The code reverse generation module obtains the code template in the benchmark operating environment where the algorithm is located, and combines it with the input of the corresponding algorithm parameters entered in the user interface to reversely generate the algorithm code and display it on the interface for the user to edit, thereby obtaining the algorithm code edited by the user;

[0024] Step 5: The user code verification module verifies whether the algorithm code manually edited by the user complies with the rules. If so, the native code is generated;

[0025] In step 6, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment to complete data reading and input, code compilation and execution, and data output and caching.

[0026] The second technical solution of the present invention is also characterized in that each algorithm in step 1 comes from a different or the same open source platform or self-developed platform, and the implementation process of the same algorithm on different platforms is included in the algorithm library; and the implementation of each algorithm in multiple benchmark operating environments has a standardized code template stored in the local Git code library;

[0027] In step 2, the input of the first process node algorithm in the data analysis process is the data source to be analyzed, and the input of the remaining process node algorithms is the output after the execution of the previous one or more process node algorithms; the process description file generated by the process construction module includes the ID of each algorithm in the process node, as well as the input and output port connection relationship between each algorithm in the data analysis process.

[0028] In step 3, the operator platform selection module selects a suitable benchmark operating environment based on the input data scale, data type, and hardware configuration of the process node algorithm, and writes the benchmark operating environment corresponding to each algorithm into the process description file.

[0029] Step 4 is specifically as follows: the code reverse generation module obtains the code template of the benchmark operating environment where the algorithm is located from the local Git code library, and combines the input of the corresponding algorithm parameters entered in the user interface to reversely generate the algorithm code corresponding to its benchmark operating environment and display it on the interface for the user to edit. The name of each algorithm code is the same as its ID in the process node. After the user edits it, the user obtains the edited algorithm code.

[0030] The user code verification module provides each algorithm with standard input test data and standard output data. The standard input test data is input into the edited algorithm code. If the format of the output data is consistent with the standard output data format, the verification is successful and the native code is generated, that is, the algorithm code edited and verified by the user; if it cannot be output or the format is inconsistent, the verification fails.

[0031] Step 6 is as follows: the process execution engine parses the process description file, loops through each process node, determines whether it meets the execution conditions, and sends the algorithm code of the process node that meets the conditions to the cluster computing host that matches its benchmark operating environment for execution;

[0032] After receiving the code sent by the process execution engine, the cluster computing host reads the code snippet and determines whether the local environment meets the code execution conditions. If not, it downloads the image package that the algorithm depends on to the local environment. The cluster computing host compiles the code locally and reads the set input data from the HBase cache database according to the node relationship set in the data analysis process. The cluster computing host executes the code locally and writes the final output data to the HBase cache database and the execution status to the Redis cache database.

[0033] In step 6, for each process description file, the process execution engine loops until all algorithm nodes in the process description file are successfully executed. The process execution engine determines whether each algorithm is successfully executed by reading the algorithm execution status written by the cluster computing host in the Redis cache database. The process execution engine obtains the data after each algorithm is executed by reading the algorithm execution results written by the algorithm node in the HBase cache database.

[0034] In step 6, the process execution engine parses the process description file and traverses the algorithm nodes in the process execution file every n seconds. If the predecessor nodes of an algorithm have been successfully executed, the algorithm will be executed in this traversal, that is, the algorithm code of the process node will be sent to the cluster computing host matching its benchmark operating environment for execution; if there is an algorithm whose predecessor node has not been successfully executed, the algorithm will not be executed in this traversal, and it will wait for the next traversal to determine whether to execute it; if there is an execution error of a node in the process, the entire analysis process is marked as an error state and the traversal ends; if all nodes in the process are successfully executed, the entire analysis process is marked as a success state and the traversal ends.

[0035] The beneficial effects of the present invention are:

[0036] The present invention realizes the interface and visualization of data analysis process construction through the process construction module, simplifies the data analysis process, and solves the problem of analysis efficiency. The algorithm platform selection module is used to realize the automatic selection of each algorithm benchmark operating environment according to the data scale, type, and hardware configuration, without the need to pre-specify the environment of each algorithm, which also improves the analysis efficiency. The code reverse generation module is used to reversely generate the benchmark platform code of the algorithm, and its validity is guaranteed by the code verification module, solving the problem of flexible adaptation of the algorithm. By integrating heterogeneous algorithms of various types and platforms into a data analysis process, and then sending different types of algorithms in the process to different types of computing nodes for execution through the process execution engine, the problem of the integration of heterogeneous algorithms is effectively solved, such as the integration of multiple types of enterprise legacy algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram of the structure of a system for constructing a big data analysis algorithm library according to the present invention;

[0038] FIG2 is a flow chart of a method for constructing a big data analysis algorithm library according to the present invention;

[0039] FIG3 is a schematic diagram of a multi-platform algorithm component set in the method for constructing a big data analysis algorithm library of the present invention;

[0040] FIG4 is a schematic diagram of a Web interface analysis process construction method for constructing a big data analysis algorithm library according to the present invention;

[0041] FIG5 is a flow chart of operator platform selection in the method for constructing a big data analysis algorithm library of the present invention;

[0042] FIG6 is a flowchart of code reverse generation in the method for constructing a big data analysis algorithm library of the present invention;

[0043] FIG7 is a flowchart of user code verification in the method for constructing a big data analysis algorithm library of the present invention;

[0044] FIG8 is an execution flow chart of the process execution engine in the method for constructing a big data analysis algorithm library of the present invention;

[0045] FIG9 is a flowchart of the execution of computing nodes in the method for constructing a big data analysis algorithm library of the present invention;

[0046] 10 is a diagram of the process of building an interface for "power consumption forecast data" in Example 4 of the present invention;

[0047] 11 is a diagram of the interface for constructing the "workpiece quality compliance prediction" process in Example 4 of the present invention;

[0048] 12 is a diagram of a configuration parameter interface of a decision tree regression algorithm in Example 4 of the present invention;

[0049] 13 is a diagram of the code editing interface after reverse code generation in Example 4 of the present invention;

[0050] 14 is a diagram of an interface after code verification fails in Example 4 of the present invention;

[0051] FIG15 is a diagram showing a feedback interface in which a process is sent to a process engine for execution and each algorithm is successfully executed in accordance with Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example 1

[0054] The structure of the big data analysis algorithm library construction system is shown in Figure 1 and includes:

[0055] The analysis process building module allows users to visually select algorithms and connect them end to end to build a complete data analysis process and generate a process description file;

[0056] The operator platform selection module selects a benchmark operating environment for each algorithm in the data analysis process built by the analysis process construction module, and writes the benchmark operating environment corresponding to each algorithm into the process description file;

[0057] The code reverse generation module generates the algorithm code of each algorithm in its benchmark operating environment, which can be manually edited by users;

[0058] User code verification module verifies whether the algorithm code manually edited by the user complies with the rules. If it does, it generates native code;

[0059] The cluster control host is deployed with a process execution engine, which sends the native code of each algorithm to the cluster computing host that matches its environment;

[0060] The cluster computing host receives the native code issued by the process execution engine, completes data reading and input, code compilation and execution, and data output and caching. Among them, cluster computing hosts include multiple categories, and each category of cluster computing hosts only deploys one benchmark operating environment. The cluster control host selects the corresponding cluster computing host based on the benchmark operating environment of the algorithm to issue native code.

[0061] Cluster computing hosts are divided into multiple types based on the underlying open source platform or language environment they are deployed on, such as Matlab, Scikit-learn, H2O, or other language platform environments. Each type can include multiple cluster computing hosts, and each cluster computing host only deploys one environment.

[0062] Example 2

[0063] The method for constructing a big data analysis algorithm library adopts the construction system of the big data analysis algorithm library in Example 1. The process is shown in Figure 2 and is specifically implemented in the following steps:

[0064] Step 1: The algorithm library initially includes various algorithms used in big data analysis, as shown in Figure 3. These algorithms include data source reading algorithms, data preprocessing algorithms, exploratory analysis algorithms, feature engineering algorithms, core analysis algorithms, parameter selection algorithms, result evaluation algorithms, and result presentation algorithms. Each algorithm comes from a different or the same open source platform or a self-developed platform, such as the Scikit-learn-based SVM algorithm and the Matlab-based random forest algorithm. The implementation of the same algorithm on different platforms is also included in the algorithm library, such as the Scikit-learn-based decision tree algorithm and the Spark-based decision tree algorithm. Each algorithm's implementation in multiple benchmark operating environments has standardized code templates stored in the local Git code repository.

[0065] In step 2, users visually edit data analysis process nodes through the analysis process building module according to their needs to form a data analysis process. Each process node corresponds to an algorithm in step 1, and a process description file is generated. As shown in Figure 4, the input of the algorithm for the first process node in the data analysis process is the data source to be analyzed, and the input of the algorithms for the remaining process nodes is the output after the execution of one or more algorithms in the previous process node. The process description file includes the ID of each algorithm in the process node and the input and output port connection relationships between the algorithms in the data analysis process.

[0066] Each algorithm defines standard input and output, that is, for multiple platforms implementing the same algorithm in the algorithm library, the input and output are the same, and the platform is the benchmark operating environment;

[0067] Step 3, as shown in Figure 5, the operator platform selection module selects a suitable benchmark operating environment based on the input data size and data type of the process node algorithm and the hardware configuration of the cluster, and writes the benchmark operating environment corresponding to each algorithm into the process description file;

[0068] Step 4, as shown in Figure 6, the code reverse generation module obtains the code template of the benchmark operating environment where the algorithm is located from the local Git code library, and combines it with the input of the corresponding algorithm parameters entered in the user interface to reversely generate the algorithm code corresponding to its benchmark operating environment and display it on the interface for the user to edit. The name of each algorithm code is the same as its ID in the process node. After the user edits it, the user obtains the edited algorithm code;

[0069] Step 5, as shown in Figure 7, the user code verification module provides standard input test data and standard output data for each algorithm. The standard input test data is input into the edited algorithm code. If the format of the output data is consistent with the standard output data format, the verification is successful, and the native code is generated, that is, the algorithm code edited and verified by the user; if the output cannot be output or the format is inconsistent, the verification fails;

[0070] Step 6, as shown in Figure 8, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment to complete data reading and input, code compilation and execution, and data output and caching. Specifically, the process execution engine parses the process description file, loops through each process node, determines whether it meets the execution conditions, and sends the algorithm code of the process node that meets the conditions to the cluster computing host that matches its benchmark operating environment for execution;

[0071] As shown in Figure 9, after the cluster computing host receives the code sent by the process execution engine, it reads the code snippet and determines whether the local environment meets the execution conditions of the code. If not, it downloads the image package that the algorithm depends on to the local environment and then continues to execute: the cluster computing host locally compiles the code and reads the set input data from the HBase cache database according to the node relationship set in the data analysis process. The cluster computing host locally executes the code and writes the final output data to the HBase cache database and the execution status to the Redis cache database. If it meets the conditions, it directly proceeds to: the cluster computing host locally compiles the code and subsequent operations.

[0072] Among them, for each process description file, the process execution engine loops until all algorithm nodes in the process description file are executed successfully. The process execution engine determines whether each algorithm is executed successfully by reading the algorithm execution status written by the cluster computing host in the Redis cache database. The process execution engine obtains the data after each algorithm is executed by reading the algorithm execution results written by the algorithm node in the HBase cache database.

[0073] Among them, when the process execution engine parses the process description file, it traverses the algorithm nodes in the process execution file every n seconds. If the predecessor nodes of a certain algorithm have been successfully executed, the algorithm will be executed in this traversal, that is, the algorithm code of the process node will be sent to the cluster computing host matching its benchmark operating environment for execution; if there is a predecessor node of a certain algorithm that has not been successfully executed, the algorithm will not be executed in this traversal, and it will wait for the next traversal to determine whether to execute it; if there is an execution error of a node in the process, the entire analysis process is marked as an error state and the traversal ends; if all nodes in the process are successfully executed, the entire analysis process is marked as a success state and the traversal ends.

[0074] When selecting an algorithm in the visual interface in step 2, users only need to focus on the algorithm's functionality, not the platform environment to which each algorithm belongs. The operator platform selection module selects the appropriate platform based on the input data size and data type (such as two-dimensional data, tensor data, signal data, etc.) of the corresponding node algorithm, as well as the cluster's hardware configuration. The operator platform selection module pre-defines several levels based on the number of data items: large-scale data (over 100 million, including 100 million); medium-scale data (between 10 million and 100 million, including 10 million); and small-scale data (under 10 million). The platform also defines the dataset types: two-dimensional data, tensor data, and signal data (also known as time series data). Based on the cluster's hardware configuration (CPU core count and memory size), the hosts are categorized into compute-intensive hosts (32 cores or more), memory-intensive hosts (128GB or more), and small-scale hosts (with less than the aforementioned CPU and memory requirements). The computing hosts in the cluster are pre-deployed with scikit-learn (a small-scale computing host), Spark (an in-memory computing host), Matlab (a computational computing host), and H2O (a host environment with multiple configurations) based on the hardware configuration. The platform then distributes large-scale, two-dimensional or tensor data to in-memory computing hosts, medium-scale data to computational computing hosts, small-scale data to small-scale computing hosts, and signal-type data to hosts deployed with the H2O environment, based on the scale or type of input data at the process nodes. For example, if a user selects the Random Forest algorithm, the algorithm library includes implementations of this algorithm on multiple platforms. For smaller data sizes, the Python Scikit-learn implementation is automatically selected, while for larger data sizes, the SparkMLlib implementation is selected. Users can also manually specify the algorithm platform.

[0075] Example 3

[0076] In this embodiment, based on the second embodiment, when the process execution engine parses the process description file, it traverses the algorithm nodes in the process execution file every 1 second.

[0077] Example 4

[0078] This embodiment uses a Web-based analysis process construction module, as shown in Figures 10 and 11, to construct two analysis processes, "electricity consumption prediction data" and "workpiece quality compliance prediction". Figures 10 and 11 are two examples of analysis processes built by the system. All algorithms supported by the platform (regardless of the language implemented) are listed in the tree menu on the left side of the interface. Dragging the tree menu node into the workspace will generate an algorithm instance. The workspace algorithms are visualized with a small square. The input and output relationship between algorithms is represented by the lines between them. Each algorithm in the figure comes from a different platform (such as scikit-learn, matlab, etc.), which are integrated together to complete the entire data analysis process. When the user selects an algorithm in the algorithm component set on the left, there is no need to manually specify which platform a single algorithm belongs to. The platform will automatically determine it.

[0079] Figures 12-14 respectively show the configuration parameter interface of the decision tree regression algorithm in the above analysis process, the code editing interface after reverse code generation, and the display after code verification fails.

[0080] Figure 15 shows the above process being sent to the process engine for execution, and the feedback interface showing that each operator has been successfully executed.

[0081] This embodiment is an interface-based implementation of the present invention. A data analysis process is constructed through a process editing interface, such as the analysis processes constructed in Figures 10 and 11. The algorithm platform selection module selects the algorithm benchmark operating platform, such as the decision tree regression algorithm shown in Figure 12, where the platform automatically selects the scikit-learn algorithm. The code reverse generation module generates algorithm code for user editing, such as Figure 13. After the user enters the key algorithm parameters, the platform generates the code for the decision tree regression algorithm in the benchmark platform scikit-learn.

[0082] The present invention solves the efficiency problem of data analysis through process-based and visual algorithm configuration; realizes the optimal platform algorithm screening through multi-platform and automated algorithm selection; realizes the combination of interface configuration and user modification through reverse code template generation, thereby improving the flexibility of algorithm use; and realizes the seamless connection of various types of algorithms in the global analysis process through distributed and cross-platform fusion methods.

Claims

1. A method for constructing a big data analysis algorithm library, characterized in that: The construction system using the big data analysis algorithm library includes: an analysis process construction module, which allows users to visually select algorithms and connect them end to end to build a complete data analysis process and generate a process description file; an operator platform selection module, which selects a benchmark operating environment for each algorithm in the data analysis process constructed by the analysis process construction module, and writes the benchmark operating environment corresponding to each algorithm into the process description file; a code reverse generation module, which generates the algorithm code of each algorithm in its benchmark operating environment, so that users can manually edit it; a user code verification module, which verifies whether the algorithm code manually edited by the user complies with the rules, and generates native code if it does; a cluster control host, which is deployed with a process execution engine, which sends the native code of each algorithm to the cluster computing host that matches its environment; the cluster computing host, which receives the native code sent by the process execution engine, completes the reading and input of data, the compilation and running of code, and the output and caching of data; Cluster computing hosts come in multiple categories, and each type of cluster computing host deploys only one benchmark operating environment. The cluster control host selects the corresponding cluster computing host based on the algorithm's benchmark operating environment and delivers native code. The specific implementation steps are as follows: Step 1: The algorithm library initially includes various algorithms used in big data analysis; Step 2: Users visually edit data analysis process nodes through the analysis process building module according to their needs to form a data analysis process. Each process node corresponds to an algorithm in step 1, and a process description file is generated; Step 3: The operator platform selection module selects a suitable benchmark operating environment for the algorithm of this process node and writes the benchmark operating environment corresponding to each algorithm into the process description file. The operator platform selection module selects a suitable benchmark operating environment based on the input data scale and data type of the algorithm of this process node and the hardware configuration of the cluster, and writes the benchmark operating environment corresponding to each algorithm into the process description file. Step 4: The code reverse generation module obtains the code template in the benchmark operating environment where the algorithm is located, and combines it with the input of the corresponding algorithm parameters entered in the user interface to reversely generate the algorithm code and display it on the interface for the user to edit, thereby obtaining the algorithm code edited by the user; Step 5: The user code verification module verifies whether the algorithm code manually edited by the user complies with the rules. If so, the native code is generated; In step 6, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment to complete data reading and input, code compilation and execution, and data output and caching.

2. The method for constructing a big data analysis algorithm library according to claim 1, characterized in that: Each algorithm in step 1 comes from a different or the same open source platform or self-developed platform, and the implementation process of the same algorithm on different platforms is included in the algorithm library; and the implementation of each algorithm in multiple benchmark operating environments has a standardized code template stored in the local Git code library; In step 2, the input of the first process node algorithm in the data analysis process is the data source to be analyzed, and the input of the remaining process node algorithms is the output of the previous or multiple process node algorithms after execution; The process description file generated by the process construction module includes the ID of each algorithm in the process node, as well as the input and output port connection relationship between each algorithm in the data analysis process.

3. The method for constructing a big data analysis algorithm library according to claim 2, characterized in that: The specific steps of step 4 are as follows: the code reverse generation module obtains the code template of the benchmark operating environment where the algorithm is located from the local Git code library, and combines the input of the corresponding algorithm parameters input in the user interface to reversely generate the algorithm code corresponding to its benchmark operating environment and display it on the interface for the user to edit. The name of each algorithm code is the same as its ID in the process node. After the user edits it, the user's edited algorithm code is obtained.

4. The method for constructing a big data analysis algorithm library according to claim 3, characterized in that: The user code verification module provides each algorithm with standard input test data and standard output data. The standard input test data is input into the edited algorithm code. If the format of the output data is consistent with the standard output data format, the verification is successful and the native code is generated, that is, the algorithm code edited and verified by the user; if it cannot be output or the format is inconsistent, the verification fails.

5. The method for constructing a big data analysis algorithm library according to claim 4, characterized in that: The step 6 is specifically as follows: The process execution engine parses the process description file, loops through each process node, determines whether it meets the execution conditions, and sends the algorithm code of the process node that meets the conditions to the cluster computing host that matches its benchmark operating environment for execution; After receiving the code sent by the process execution engine, the cluster computing host reads the code snippet and determines whether the local environment meets the execution conditions of the code. If not, it downloads the image package that the algorithm depends on to the local environment. The cluster computing host compiles the code locally and reads the set input data from the HBase cache database according to the node relationship set in the data analysis process. The cluster computing host executes the code locally and writes the final output data to the HBase cache database and the execution status to the Redis cache database.

6. The method for constructing a big data analysis algorithm library according to claim 5, characterized in that: In step 6, for each process description file, the process execution engine loops until all algorithm nodes in the process description file are successfully executed. The process execution engine determines whether each algorithm is successfully executed by reading the algorithm execution status written by the cluster computing host in the Redis cache database. The process execution engine obtains the data after each algorithm is executed by reading the algorithm execution results written by the algorithm node in the HBase cache database.

7. The method for constructing a big data analysis algorithm library according to claim 6, characterized in that: In step 6, the process execution engine parses the process description file and traverses the algorithm nodes in the process execution file every n seconds. If all the preceding nodes of an algorithm have been successfully executed, the algorithm will be executed in this traversal, that is, the algorithm code of the process node will be sent to the cluster computing host that matches its benchmark operating environment for execution; If a certain algorithm has a predecessor node that has not been successfully executed, the algorithm will not be executed in this traversal, and the next traversal will be waited for to determine whether to execute it; if there is an execution error in a node in the process, the entire analysis process is marked as an error state and the traversal ends; if all nodes in the process are successfully executed, the entire analysis process is marked as a success state and the traversal ends.

Images