Interactive automated big data analysis system and method

Abstract

The embodiment of the invention provides an interactive automated big data analysis system and method. The interactive automated big data analysis system includes a data receiving module, an operatormanagement module, an analysis model generation module, a process analysis and optimization module and a task scheduling execution module. First, the system supports an interactive function with strong ease of use; second, the system provides users with functions such as batch build process, differentiated multiplexing process, and dynamic planning process through flow control operators, which greatly simplifies the building steps of complex analysis processes. In particular, the process analysis and optimization module is used to perform algorithm selection for each analysis process, users donot have to consider the algorithm implementation problem, and the users' use threshold in the data analysis technical aspect is greatly reduced; in addition, the process analysis and optimization module is used to perform hyperparameter selection for processes needing to undergo hyperparameter optimization. The system automatically selects optimal execution hyperparameters for tasks with uncertain parameters, making the construction and execution of the analysis process more concise and efficient.

Description

technical field [0001] Embodiments of the present invention relate to the technical field of big data analysis, and more specifically, relate to an interactive big data analysis system and method. Background technique [0002] In recent years, a large number of technologies and products have been developed in the field of big data, which have become effective means for big data acquisition, storage, processing analysis or visualization. However, there are high technical barriers to using these technologies and products. How to organize big data technologies in a standardized form and provide users with convenient and efficient usage methods has become a huge challenge. [0003] A series of big data analysis products based on cloud platforms, represented by many large Internet companies, have emerged as the times require. These products provide relatively efficient and relatively convenient means for the organization of big data analysis tasks by virtue of their abundant clo...

AI Technical Summary

Problems solved by technology

However, the data analysts of enterprises are often proficient in domain knowledge and business background, but not very familiar with the technical details of data analysis, resulting in these big data analysis products still have a high technical threshold

Especially in today's rapid development of data analysis technology, most enterprises cannot bear the huge labor cost of mastering various data analysis technologies at the same time

In addition, in practical applications, it is often necessary to train batches of similar models according to different conditions, and existing platforms and products cannot efficiently support this scenario

Images