Construction system and method for big data analysis algorithm library

Abstract

Disclosed in the present invention is a construction system for a big data analysis algorithm library, comprising: an analysis process construction module, an operator platform selection module, a code reverse generation module, a user code verification module, a cluster control host, and a cluster computing host. Cross-platform and cross-language algorithm fusion can be realized, and data analysis is realized in a complete and heterogeneous data analysis flow. Also disclosed in the present invention is a construction method for a big data analysis algorithm library.

Description

[0002] With the development of big data ecological software, big data analysis algorithms based on big data technology and centered on data processing and business analysis have also been widely used; therefore, building an algorithm library for industry big data analysis is one of the necessary elements to promote the development of the big data industry.

[0003] Most traditional big data algorithm libraries are designed to enrich the platform algorithms. Still, they cannot fully meet the above requirements in terms of the heterogeneity of cross-platform algorithms, multi-language compatibility, algorithm selection, and flexible use of algorithms. Commonly used data analysis algorithms are based on unified language and environment development, for example, the small-scale data analysis algorithm library represented by Scikit-learn, the large-scale data analysis algorithm library represented by Spark MLlib, the data statistical analysis algorithm library represented by MATLAB, the deep learning algorithm framework represented by Pytorch and Tensorflow, and the signal data processing algorithm library represented by H2O. However, in the actual industry data analysis, it usually requires multiple algorithms on multiple platforms to build a complete data analysis process, which requires users to select the platform according to the business, and then select the algorithm according to the platform. If the multiple algorithms required by the process exist in multiple platforms, only manual coding can be used to achieve the environmental consistency of these algorithms. In addition, traditional enterprises have accumulated a large number of selfdeveloped algorithms to solve the problem of industry data analysis in the process of development, these industry-oriented special algorithms may be based on any programming environment, and need to reconstruct the code to adapt to the entire analysis process when integrating into the algorithm library. All of the above undoubtedly bring difficulties to the establishment of an enterprise-oriented big data analysis algorithm library.

[0004] In summary, the existing big data analysis algorithm library or big data analysis platform mainly has the following problems in the industry-oriented data analysis scenario:

[0005] Analysis efficiency: Traditional data analysis requires cross-domain experts to spend a lot of time on business communication and algorithm debugging, and the threshold is high, which causes a lot of inconvenience to analysis modeling and design.

[0006] Algorithm selection problem: In traditional data analysis, it is impossible to combine the advantages of each open source platform algorithm and automatically select the algorithm of a specific platform; it requires a lot of manual intervention to select the appropriate platform and further select the appropriate algorithm.

[0007] Algorithm adaptation problem: The traditional data analysis platform provides a complete interface configuration so that users cannot complete the customized modification of the code, and the form of interface calls is difficult for beginners to use, lacking the flexibility of the data analysis process.

[0008] Algorithm fusion problem: The traditional analysis algorithm platform cannot support the technical interoperability and seamless connection between the new platform algorithm and the enterprise historical algorithm, and thus cannot realize the extensive compatibility and sustainable evolution of the industry’s algorithm library. SUMMARY

[0009] The purpose of the invention is to provide a construction system of a big data analysis algorithm library, the construction system can realize cross-platform and crosslanguage algorithm fusion, and realize data analysis in a complete and heterogeneous data analysis flow.

[0010] Another purpose of the invention is to provide a construction method of a big data analysis algorithm library.

[0011] The technical scheme of the invention is the construction system of the big data analysis algorithm library, including:

[0012] analysis process construction module, the user visually selects the algorithm and connects it to build a complete data analysis process, and a process description file is generated;

[0013] algorithm platform selection module, the algorithm platform selection module selects the benchmark running environment for each algorithm in the data analysis process constructed by the analysis process construction module, and the benchmark running environment corresponding to each algorithm is written to the process description file;

[0014] code reverse generation module, the code reverse generation module generates the algorithm code of each algorithm in its benchmark running environment, so that the user can manually edit;

[0015] user code verification module, the user code verification module verifies whether the algorithm code manually edited by the user conforms to the rules, and if it does, the native code is generated;

[0016] cluster control host, the cluster control host deploys a process execution engine, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment;

[0017] cluster computing host, the cluster computing host receives the native code issued by the process execution engine, reads and inputs the data, compiles and runs the code, and outputs and caches the data.

[0018] The invention is also characterized in that the cluster computing host includes multiple types, each type of cluster computing host only deploys a benchmark running environment, and the cluster control host selects the corresponding cluster computing host to distribute the native code according to the benchmark running environment of the algorithm.

[0019] Another technical scheme adopted by the invention is the construction method of the big data analysis algorithm library, which adopts the construction system of the big data analysis algorithm library, the method is implemented in the following steps:

[0020] Step 1, the algorithm library initially includes various algorithms used in big data analysis;

[0021] Step 2, the user visually edits the data analysis process nodes through the analysis process construction module according to the requirements, and forms the data analysis process, each process node corresponds to an algorithm in Step 1 and generates a process description file.

[0022] Step 3, the algorithm platform selection module selects the appropriate benchmark running environment for the algorithm of the process node, and writes the benchmark running environment corresponding to each algorithm into the process description file;

[0023] Step 4, the code reverse generation module obtains the code template in the benchmark running environment where the algorithm is located, and combines the input of the corresponding algorithm parameters input by the user interface to reversely generate the algorithm code and display it on the interface for user editing to obtain the user-edited algorithm code;

[0024] Step 5, the user code verification module verifies the algorithm code manually edited by the user and whether it conforms to the rules; if so, the native code is generated;

[0025] Step 6, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment, completes the reading and input of data, the compilation and operation of code, and the output and cache of data.

[0026] The second technical scheme of the invention is also characterized in that each algorithm in Step 1 comes from different or the same open source platform or selfdeveloped platform, and the implementation process of the same algorithm on different platforms is included in the algorithm library, and the implementation of each algorithm in multiple benchmark running environments has a standardized code template stored in the local Git code library;

[0027] in Step 2, the input of the first process node algorithm in the data analysis process is the data source to be analyzed, and the input of the remaining process node algorithms is the output of the previous one or more process node algorithms, the process description file generated by the process construction module includes the ID of each algorithm in the process node and the input and output port connection relationship between each algorithm in the data analysis process.

[0028] In Step 3, the algorithm platform selection module in Step 3 selects the appropriate benchmark running environment according to the input data size, data type, and cluster hardware configuration of the process node algorithm, and writes the benchmark running environment corresponding to each algorithm into the process description file.

[0029] Step 4 is as follows: the code reverse generation module obtains the code template in the benchmark running environment where the algorithm is located from the local Git code library, and combines the input of the parameters of the corresponding algorithm input by the user interface to reversely generate the algorithm code corresponding to the benchmark running environment of the algorithm and display it on the interface for user editing, and the naming of each algorithm code is the same as its ID in the process node, after user editing, the algorithm code after user editing is obtained.

[0030] The user code verification module provides standard input test data and standard output data for each algorithm, the standard input test data is input into the edited algorithm code, if the format of the output data is consistent with the standard output data format, the verification is successful, and the native code is generated, that is, the user-edited and validated algorithm code; if it is unable to output or the format is inconsistent, the verification fails.

[0031] Step 6 is as follows: the process execution engine parses the process description file, iterates through each process node to determine whether it meets the execution conditions, and sends the algorithm code of the process node that meets the conditions to the cluster computing host that matches its benchmark running environment;

[0032] after the cluster computing host receives the code issued by the process execution engine, it reads the code fragment and determines whether the local environment has satisfied the execution condition of the code, if it does not satisfy, the mirror package that the algorithm relies on is downloaded to the local, the cluster computing host compiles the code locally, and reads the set input data from the HBase cache database according to the node relationship set by the data analysis process, the cluster calculates the local execution code of the host, and then writes the final output data to the HBase cache database, and writes the execution state to the Redis cache database.

[0033] In Step 6, for each process description file, the process execution engine loops until all the algorithm nodes in the process description file are successfully executed, the process execution engine judges whether each algorithm is successfully executed by reading the algorithm execution status written by the cluster computing host in the Redis cache database, the process execution engine obtains the data after each algorithm is executed by reading the algorithm execution results written by the algorithm nodes in the HBase cache database.

[0034] In Step 6, the process execution engine parses the process description file, and it traverses the algorithm nodes in the process execution file every n seconds, if the pre-join nodes of an algorithm have been successfully executed, the algorithm will be executed in this traversal, that is, the algorithm code of the process node is sent to the cluster computing host matching its benchmark running environment; if an algorithm has a front node that has not been successfully executed, the algorithm is not executed in this traversal, and the next traversal is waited to determine whether it is executed; if there is an error in the execution of the node in the process, the entire analysis process is marked as an error state and the traversal is ended; if all nodes in the process are successfully executed, the entire analysis process is marked as a successful state and the traversal is ended.

[0035] The beneficial effects of the invention are:

[0036] The invention realizes the interface and visual data analysis process construction through the process construction module, simplifies the data analysis process, and solves the problem of analysis efficiency. Through the algorithm platform selection module, the selection of the benchmark operating environment of each algorithm is automatically selected according to the data size, type, and hardware configuration. There is no need to specify the environment of each algorithm in advance, which also improves the analysis efficiency. Through the code reverse generation module, the benchmark platform code of the algorithm is reversely generated, and its effectiveness is guaranteed through the code verification module, which solves the problem of flexible adaptation of the algorithm. By integrating the heterogeneous algorithms of various types and platforms into a data analysis process, and then sending different types of algorithms in the process to different types of computing nodes through the process execution engine, the integration problem of heterogeneous algorithms is effectively solved, such as the integration of multi-type enterprise history legacy algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0037] FIG. 1 is a structural diagram of the construction system of the big data analysis algorithm library of the invention;

[0038] FIG. 2 is a flow chart of the construction method of the big data analysis algorithm library of the invention;

[0039] FIG. 3 is a schematic diagram of the multi-platform algorithm component set in the construction method of the big data analysis algorithm library of the invention;

[0040] FIG. 4 is a schematic diagram of the Web interface analysis process in the construction method of the big data analysis algorithm library of the invention;

[0041] FIG. 5 is a flow chart of the algorithm platform selection in the construction method of the big data analysis algorithm library of the invention;

[0042] FIG. 6 is a flow chart of code reverse generation in the construction method of the big data analysis algorithm library of the invention;

[0043] FIG. 7 is a flow chart of user code verification in the construction method of the big data analysis algorithm library of the invention;

[0044] FIG. 8 is an execution flow chart of the process execution engine in the construction method of the big data analysis algorithm library of the invention;

[0045] FIG. 9 is an execution flow chart of the calculation node in the construction method of the big data analysis algorithm library of the invention;

[0046] FIG. 10 is an interface diagram of the process construction of'power consumption prediction data' in Embodiment 4 of the invention.

[0047] FIG. 11 is an interface diagram of the process construction of'workpiece quality coincidence rate prediction' in Embodiment 4 of the invention.

[0048] FIG. 12 is a configuration parameter interface diagram of the decision tree regression algorithm in Embodiment 4 of the invention.

[0049] FIG. 13 is the code editing interface diagram after reverse code generation in Embodiment 4 of the invention.

[0050] FIG. 14 is an interface diagram after the failure of code verification in Embodiment 4 of the invention.

[0051] FIG. 15 is a feedback interface diagram of the process issued to the process engine and the successful execution of each algorithm in Embodiment 4 of the invention.

[0052] The following is a detailed description of the invention in combination with the attached diagram and the specific implementation method.

[0053] Embodiment 1

[0054] The construction system of the big data analysis algorithm library, the structure is shown in FIG. 1, including:

[0055] analysis process construction module, the user visually selects the algorithm and connects it to build a complete data analysis process, and a process description file is generated;

[0056] algorithm platform selection module, the algorithm platform selection module selects the benchmark running environment for each algorithm in the data analysis process constructed by the analysis process construction module, and the benchmark running environment corresponding to each algorithm is written to the process description file;

[0057] code reverse generation module, the code reverse generation module generates the algorithm code of each algorithm in its benchmark running environment, so that the user can manually edit;

[0058] user code verification module, the user code verification module verifies whether the algorithm code manually edited by the user conforms to the rules, and if it does, the native code is generated;

[0059] cluster control host, the cluster control host deploys a process execution engine, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment;

[0060] cluster computing host, the cluster computing host receives the native code issued by the process execution engine, reads and inputs the data, compiles and runs the code, and outputs and caches the data, where the cluster computing host includes multiple types, each type of cluster computing host only deploys a benchmark operating environment, and the cluster control host selects the corresponding cluster computing host to issue the native code according to the benchmark operating environment of the algorithm.

[0061] Cluster computing hosts are divided into various types according to the basic open source platform or language environment they deploy, such as Matlab, Scikit-leam, H2O, or other language platform environments, each type can contain multiple cluster computing hosts, and each cluster computing host only deploys one environment.

[0062] Embodiment 2

[0063] The construction method of the big data analysis algorithm library adopts the construction system of the big data analysis algorithm library in Embodiment 1, the process is shown in FIG. 2, which is implemented in the following steps:

[0064] Step 1, the algorithm library initially includes various algorithms used in big data analysis, as shown in FIG. 3, including data source reading algorithm, data preprocessing algorithm, exploratory analysis algorithm, feature engineering algorithm, core analysis algorithm, parameter selection algorithm, result evaluation algorithm, result display algorithm; each algorithm comes from different or the same open source platform or selfdeveloped platform, such as the SVM algorithm based on Scikit-leam and the random forest algorithm based on Matlab, and the implementation process of the same algorithm on different platforms is included in the algorithm library, such as the decision tree algorithm based on Scikit-learn and the decision tree algorithm based on Spark. And the implementation of each algorithm in multiple benchmark running environments has a standardized code template stored in the local Git code library;

[0065] In Step 2, the user visually edits the data analysis process nodes through the analysis process construction module according to the requirements, and forms the data analysis process, each process node corresponds to an algorithm in Step 1 and generates a process description file; as shown in FIG. 4, the input of the first process node algorithm in the data analysis process is the data source to be analyzed, and the input of the remaining process node algorithms is the output of the previous one or more process node algorithms; the process description file includes the ID of each algorithm in the process node and the input and output port connection relationship between each algorithm in the data analysis process.

[0066] Each algorithm defines the standard input and output, that is, for the multiplatform implementation of the same algorithm in the algorithm library, the input and output are the same, and the platform is the benchmark running environment.

[0067] Step 3, as shown in FIG. 5, the algorithm platform selection module selects the appropriate benchmark running environment according to the input data size, data type, and cluster hardware configuration of the process node algorithm, and writes the benchmark running environment corresponding to each algorithm to the process description file.

[0068] In Step 4, as shown in FIG. 6, the code reverse generation module obtains the code template in the benchmark running environment where the algorithm is located from the local Git code library, and combines the input of the parameters of the corresponding algorithm input by the user interface to reversely generate the algorithm code corresponding to its benchmark running environment, the algorithm code is displayed on the interface for user editing, and the name of each algorithm code is the same as its ID in the process node, after user editing, the algorithm code after user editing is obtained;

[0069] Step 5, as shown in FIG. 7, the user code verification module provides standard input test data and standard output data for each algorithm, and inputs the standard input test data into the edited algorithm code, if the format of the output data is consistent with the standard output data format, the verification is successful, and the native code is generated, that is, the user-edited and validated algorithm code; if the output cannot be output or the format is inconsistent, the verification fails;

[0070] Step 6, as shown in FIG. 8, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment, completes the reading and input of data, the compilation and operation of code, and the output and cache of data, specifically, the process execution engine parses the process description file, iterates through each process node, determines whether it meets the execution conditions, and sends the algorithm code of the process node that meets the conditions to the cluster computing host that matches its benchmark running environment.

[0071] As shown in FIG. 9, after receiving the code issued by the process execution engine, the cluster computing host reads the code fragment and determines whether the local environment has satisfied the execution condition of the code. If not, download the mirror package that the algorithm relies on to the local, and then continue to execute: the cluster computing host compiles the code locally, and reads the set input data from the HBase cache database according to the node relationship set by the data analysis process, the cluster calculates the local execution code of the host, and then writes the final output data to the HBase cache database, and writes the execution state to the Redis cache database, if satisfied, directly: the cluster computes the host’s local compiled code and subsequent operations.

[0072] Where for each process description file, the process execution engine loops until all the algorithm nodes in the process description file are successfully executed, the process execution engine determines whether each algorithm is successfully executed by reading the algorithm execution status written by the cluster computing host in the Redis cache database, the process execution engine obtains the data after each algorithm is executed by reading the algorithm execution results written by the algorithm nodes in the HBase cache database.

[0073] When the process execution engine parses the process description file, it traverses the algorithm nodes in the process execution file every n seconds, if the pre-join node of an algorithm has been successfully executed, the traversal will execute the algorithm, that is, the algorithm code of the process node is sent to the cluster computing host matched to its benchmark running environment for execution; if an algorithm has a front node that has not been successfully executed, the algorithm is not executed in this traversal, and the next traversal is waited to determine whether it is executed; if there is an error in the execution of the node in the process, the entire analysis process is marked as an error state and the traversal is ended; if all nodes in the process are successfully executed, the entire analysis process is marked as a successful state and the traversal is ended.

[0074] The user only needs to pay attention to the function of the algorithm when selecting the algorithm in the visual interface of Step 2, without paying attention to the platform environment of each algorithm, the algorithm platform selection module selects the appropriate platform according to the input data size, data type (such as two-dimensional data, tensor data, signal data, etc.) of the corresponding node algorithm and the hardware configuration of the cluster. When the algorithm platform selection module selects the platform, several grades are defined in advance according to the number of data items: large-scale data includes more than 100 million items; the medium-scale data is within 10 million to 100 million, including 10 million; small-scale data is less than 10 million, and the types of data sets are defined: two-dimensional data, tensor data, and signal data (also known as time series data). According to the hardware configuration of the cluster computing host (CPU core number and memory size), the computing host is divided into computing host (32 core and above), memory computing host (128G and above), and small-scale computing host (CPU and memory are less than the above value). The computing hosts in the cluster have pre-deployed scikit-leam (small-scale computing host), Spark (memory computing host), Matlab (computing host), and H2O (multiconfigured host environment) according to the hardware configuration. According to the size of the input data or the type of data, the platform sends the large-scale data in the two-dimensional or tensor data to the memory computing host, the medium-scale data to the computing host, the small-scale data to the small-scale computing host, and the signal type data to the host deployed in the H2O environment. For example, the user selects the random forest algorithm, and the algorithm library contains the implementation of the algorithm on a variety of platforms. If the data size is small, the implementation in Python Scikit-learn is automatically selected; if the data size is large, the implementation in SparkMLlib is selected, and the user can also manually specify the algorithm platform.

[0075] Embodiment 3

[0076] Based on Embodiment 2, when the process execution engine parses the process description file, it traverses the algorithm nodes in the process execution file every 1 second.

[0077] Embodiment 4

[0078] This embodiment constructs two analysis processes of 'power consumption prediction data' and 'workpiece quality coincidence rate prediction' through the Webbased analysis process construction module, as shown in FIG. 10 and FIG. 11. FIG. 10 and FIG. 11 are two examples of the two analysis processes constructed through the system. All the algorithms supported by the platform (no matter what language is implemented) are listed in the tree menu on the left side of the interface, the tree menu node is dragged into the workspace to generate an algorithm instance, and the workspace algorithm is visually represented by a small square. The input-output relationship between the algorithm and the algorithm is represented by the connection between them. Each algorithm in the figure is derived from different platforms (such as scikit-learn, matlab, etc.), and is integrated together to complete the entire data analysis process. When the user selects the algorithm in the left algorithm component set, it does not need to manually specify which platform a single algorithm belongs to, which is automatically determined by the platform.

[0079] FIG. 12-FIG. 14 are the configuration parameter interface of the decision tree regression algorithm in the above analysis process, the code editing interface after reverse code generation, and the display after code verification failure.

[0080] FIG. 15 shows that the above process is sent to the process engine for execution, and the feedback interface for the successful execution of each algorithm is obtained.

[0081] This embodiment is the realization of the interface of the invention. The data analysis process is constructed through the process editing interface, such as the analysis process constructed in FIG. 10 and FIG. 11. Through the algorithm platform selection module to select the algorithm benchmark running platform, such as the decision tree regression algorithm shown in FIG. 12, the platform automatically selects the algorithm in scikit-learn; the algorithm code is generated by the code reverse generation module for user editing. For example, after the user fills in the key parameters of the algorithm in FIG. 13, the platform generates the code of the decision tree regression algorithm in the benchmark platform scikit-learn.

[0082] The invention solves the efficiency problem of data analysis through process and visual algorithm configuration; the optimal platform algorithm screening is realized through multi-platform and automatic algorithm selection; the combination of interface configuration and user modification is realized through the generation of reverse code template to improve the flexibility of the algorithm; the seamless connection of various types of algorithms in the global analysis process is realized through the distributed and cross-platform fusion method.

Claims

1. A construction method of a big data analysis algorithm library, wherein the method adopts a construction system of the big data analysis algorithm library, including:analysis process construction module, the user visually select the algorithm and connect it to build a complete data analysis process, and a process description file is generated; algorithm platform selection module, the algorithm platform selection module selects the benchmark running environment for each algorithm in the data analysis process constructed by the analysis process construction module, and the benchmark running environment corresponding to each algorithm is written to the process description file; code reverse generation module, the code reverse generation module generates the algorithm code of each algorithm in its benchmark running environment, so that the user can manually edit; user code verification module, the user code verification module verifies whether the algorithm code manually edited by the user conforms to the rules, and if it does, the native code is generated; cluster control host, the cluster control host deploys a process execution engine, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment; cluster computing host, the cluster computing host receives the native code issued by the process execution engine, reads and inputs the data, compiles and runs the code, and outputs and caches the data;the cluster computing host includes multiple types, each type of cluster computing host only deploys a benchmark running environment, and the cluster control host selects the corresponding cluster computing host to distribute the native code according to the benchmark running environment of the algorithm;it is implemented in the following steps:Step 1, the algorithm library initially includes various algorithms used in big data analysis;Step 2, the user visually edits the data analysis process nodes through the analysis process construction module according to the requirements, and forms the data analysis process, each process node corresponds to an algorithm in Step 1 and generates a processdescription file;Step 3, the algorithm platform selection module selects the appropriate benchmark running environment for the algorithm of the process node, and writes the benchmark running environment corresponding to each algorithm into the process description file; where the algorithm platform selection module selects the appropriate benchmark running environment according to the input data size, data type and cluster hardware configuration of the process node algorithm, and writes the benchmark running environment corresponding to each algorithm into the process description file;Step 4, the code reverse generation module obtains the code template in the benchmark running environment where the algorithm is located, and combines the input of the corresponding algorithm parameters input by the user interface to reversely generate the algorithm code and display it on the interface for user editing to obtain the user-edited algorithm code;Step 5, the user code verification module verifies the algorithm code manually edited by the user and whether it conforms to the rules; if so, the native code is generated;Step 6, the process execution engine sends the native code of each algorithm to the cluster computing host that matches its environment, completes the reading and input of data, the compilation and operation of code, and the output and cache of data.

2. The construction method of the big data analysis algorithm library according to claim 1, wherein each algorithm in Step 1 comes from different or the same open source platform or self-developed platform, and the implementation process of the same algorithm on different platforms is included in the algorithm library, and the implementation of each algorithm in multiple benchmark running environments has a standardized code template stored in the local Git code library;in Step 2, the input of the first process node algorithm in the data analysis process is the data source to be analyzed, and the input of the remaining process node algorithms is the output of the previous one or more process node algorithms, the process description file generated by the process construction module includes the ID of each algorithm in the process node and the input and output port connection relationship between each algorithm in the data analysis process.

3. The construction method of the big data analysis algorithm library according to claim 2, wherein Step 4 is as follows: the code reverse generation module obtains the code template in the benchmark running environment where the algorithm is located from the local Git code library, and combines the input of the parameters of the corresponding algorithm input by the user interface to reversely generate the algorithm code corresponding to the benchmark running environment of the algorithm and display it on the interface for user editing, and the naming of each algorithm code is the same as its ID in the process node, after user editing, the algorithm code after user editing is obtained.

4. The construction method of the big data analysis algorithm library according to claim 3, wherein the user code verification module provides standard input test data and standard output data for each algorithm, the standard input test data is input into the edited algorithm code, if the format of the output data is consistent with the standard output data format, the verification is successful, and the native code is generated, that is, the user-edited and validated algorithm code; if it is unable to output or the format is inconsistent, the verification fails.

5. The construction method of the big data analysis algorithm library according to claim 4, wherein Step 6 is as follows: the process execution engine parses the process description file, iterates through each process node to determine whether it meets the execution conditions, and sends the algorithm code of the process node that meets the conditions to the cluster computing host that matches its benchmark running environment;after the cluster computing host receives the code issued by the process execution engine, it reads the code fragment and determines whether the local environment has satisfied the execution condition of the code, if it does not satisfy, the mirror package that the algorithm relies on is downloaded to the local, the cluster computing host compiles the code locally, and reads the set input data from the HBase cache database according to the node relationship set by the data analysis process, the cluster calculates the local execution code of the host, and then writes the final output data to the HBase cache database, and writes the execution state to the Redis cache database.

6. The construction method of the big data analysis algorithm library according to claim 5, wherein in Step 6, for each process description file, the process execution engineloops until all the algorithm nodes in the process description file are successfully executed, the process execution engine judges whether each algorithm is successfully executed by reading the algorithm execution status written by the cluster computing host in the Redis cache database, the process execution engine obtains the data after each algorithm is executed by reading the algorithm execution results written by the algorithm nodes in the HBase cache database.

7. The construction method of the big data analysis algorithm library according to claim 6, wherein in Step 6, the process execution engine parses the process description file, and it traverses the algorithm nodes in the process execution file every n seconds, if the pre-join nodes of an algorithm have been successfully executed, the algorithm will be executed in this traversal, that is, the algorithm code of the process node is sent to the cluster computing host matching its benchmark running environment; if an algorithm has a front node that has not been successfully executed, the algorithm is not executed in this traversal, and the next traversal is waited to determine whether it is executed; if there is an error in the execution of the node in the process, the entire analysis process is marked as an error state and the traversal is ended; if all nodes in the process are successfully executed, the entire analysis process is marked as a successful state and the traversal is ended.

Images