Data analysis system and data analysis method

Abstract

The invention discloses a data analysis system and a data analysis method. The data analysis system comprises a to-be-scheduled task generation module, a to-be-scheduled task storage module, a task scheduling module and a task processing module, wherein the to-be-scheduled task generation module is used for generating collected data into tasks to be scheduled according to pre-defined task parameters; the to-be-scheduled task storage module is used for storing the generated tasks to be scheduled; the task scheduling module is used for loading the tasks to be scheduled and calling the corresponding task processing module according to task type; the task processing module is used for generating a corresponding Hive structured query language (SQL) sentence according to an analysis requirement in the tasks, and sending the corresponding Hive SQL sentence to a Hadoop-based data warehouse server; and after data which are returned by the server are received, accomplishing data analysis of the tasks. The data analysis is performed on the bottom layer of the data analysis system by using a Hadoop system, and the tasks can be comprehensively managed on the top layer of the data analysis system by using the task scheduling module, so a data analysis flow can be simplified by using the Hadoop system, and a system for conveniently scheduling and managing the tasks is provided.

Description

technical field [0001] The invention relates to the technical field of data analysis, in particular to a data analysis system and a data analysis method. Background technique [0002] In recent years, due to the continuous growth of data in the Internet field, companies are faced with the need to process massive amounts of data. The data analysis within the department is mainly for the operation and maintenance services of various departments of the company. The data analysis is mainly based on the logs generated by the servers (Server) of each department, such as apache logs, nginx logs, etc. The purpose is to analyze user access and data based on these logs. The traffic has clear quantitative data in the time dimension, product line dimension, domain name dimension, etc., so as to provide suggestions for the company's server operation and maintenance management, traffic allocation, and forecasting. [0003] For example, to collect the nginx logs of a certain product line,...

AI Technical Summary

Problems solved by technology

[0007] However, as the scale of the website and the number of users continue to grow, the amount of data increases sharply. In the data analysis process of the existing technology, specific analysis results are obtained through complex table splitting, table splitting, and merging. The technical implementation process is complicated and requires specialized of maintenance personnel to maintain

Moreover, if new business analysis requirements need to be added, new analysis scripts need to be added, which is not conducive to expansion

[0008] To sum up, the data analysis methods in the prior art require complex operations such as table splitting, table splitting, and merging on relational databases, so the implementation process is complicated and difficult to maintain; moreover, the data analysis methods in the prior art are not Conducive to expanding new business analysis requirements

Images