Hadoop based method for analyzing large-scale social network and analysis platform thereof

Abstract

The present invention provides a Hadoop based method for analyzing a large-scale social network and an analysis platform thereof. The method comprises the following steps: 1), acquiring raw data on a social network, and storing the raw data; 2), performing unification processing on the raw data, so that a data file in a fixed format is generated by using the raw data; 3), preprocessing the data file to convert the data file into a data format of an HNAP system, and then establishing a data model that supports a graph theory model and a distributed environment; 4), using an algorithm in an algorithm library of the HNAP system to perform social network analysis on data of the data model in the step 3), integrating output results of the analysis, and generating a document file. According to the method and the platform provided by the present invention, the raw data is processed by using a distributed data processing and analysis tool, so that the data file in the fixed format is generated by using the raw data, and an analysis result is acquired through distributed computing, thereby solving the problem that a processing capability of a single core is insufficient.

Description

technical field [0001] The invention relates to the technical field of data batch processing, in particular to a Hadoop-based large-scale social network analysis method and an analysis platform thereof. Background technique [0002] The rise of social networks generates massive user data, which provides conditions for extracting further useful information and mining potential business opportunities. With the continuous expansion of the scale of the Internet, the scale of user data continues to increase, making the traditional single-core processing model unsuitable for processing massive data. They often have problems such as insufficient storage and processing capabilities when processing massive data, which makes The analysis of large-scale data becomes more difficult. At the same time, the existing distributed processing platforms are not all open source, and the provided algorithms often have limitations in analysis capabilities. [0003] At present, most big data analy...

AI Technical Summary

Problems solved by technology

With the continuous expansion of the scale of the Internet, the scale of user data continues to increase, making the traditional single-core processing model unsuitable for processing massive data. They often have problems such as insufficient storage and processing capabilities when processing massive data, which makes The analysis of large-scale data becomes more difficult. At the same time, the existing distributed processing platforms are not all open source, and the provided algorithms often have limitations in analysis capabilities.

[0003] At present, most big data analysis platforms operate under single-core conditions and rely on the algorithm itself for analysis. When processing massive data, problems such as insufficient storage and processing capabilities often occur, and system resources are not fully utilized. Efficiency not tall

At the same time, the existing distributed processing platforms are not all open source, and the provided algorithms often have limitations in analysis capabilities, and cannot fully support social network analysis, which brings difficulties to social network analysis.

And the existing big data platform does not have a balance of hardware processing capabilities and algorithm analysis capabilities

Images