Machine learning method and device, and big data platform

Abstract

The invention discloses a machine learning device, a machine learning method based on the machine learning device, and a big data platform using the machine learning device and the machine learning method thereof. The machine learning device comprises a user-defined process module, a configuration module, a database and an event server, wherein the user-defined process module comprises a logic module, the logic module can receive an executable file contained in a request initiated by a user, and can be called by the event server; and the database binds a front-end development application with the executable file through a configuration file written by the configuration module. According to the machine learning method, the machine learning device and the big data platform, the service logic component is completed by means of the user-defined process module, the adaptability and universality of the machine learning method, the machine learning device and the big data platform for various application scenarios are realized, efficient operation of data mining and machine learning involved in the development process of standardized big data are achieved, the development process of the standardized big data is simplified, and the development and deployment efficiency of the standardized big data is improved.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a machine learning method, a machine learning device, and a big data platform based on the machine learning device. Background technique [0002] Spark is Databricks' open source big data computing processing engine. It became the top Apache project in 2010. Its core computing is elastic distributed data set (RDD), which provides a richer MapReduce model than Hadoop, and can quickly process data sets in memory. Iterative computing supports complex machine learning algorithms and graph theory algorithms. [0003] The machine learning methods in the prior art are as follows. [0004] First, perform step 1) collection of raw data: the data producer will generate various types of data, such as log files, image data, text data, etc., and the quality of the data will occur with the inappropriate behavior of the user or some problems in the system There is a lot of noise data...

AI Technical Summary

Problems solved by technology

Therefore, the technical solution of the machine learning algorithm model based on spark big data in the existing technology is very complicated, and the technical level of the design covers distributed computing, architecture deployment, model computing, data development and other aspects, and it takes a lot of manpower and material resources to complete

Each process requires frequent mutual access operations with the file system, which greatly reduces the performance of the entire system, resulting in reduced reliability of modeling, prediction, and applications; more importantly, it will lead to programming flexibility, ease of maintenance, and The reusability of code or components is greatly affected, resulting in poor user experience

Images