Parallel Data Analysis Methods

Abstract

The invention provides a parallel data analysis method. The method comprises the following steps: establishing a Map / Reduce parallel computing environment for text data analysis; Starting a text mining process on each node of the business set; The parallel processes running independently on each node are organized into parallel programs. The invention provides a parallel data analysis method, which improves the parallel framework of MAPRUDUCE from the angles of process management and buffer management, and better meets the needs of text data mining. Aiming at the irregularity of social text, semantic vectors are used to represent and analyze the text data effectively, which is suitable for the analysis and calculation of social text mining of various scales.

Description

technical field [0001] The invention relates to big data mining, in particular to a parallel data analysis method. Background technique [0002] Big data, especially social network data, contains huge commercial and social value. Effective management and utilization of these data and the value mining of data will have a huge impact on enterprises and individuals. On the other hand, while big data brings new development opportunities, it also brings many technical challenges. Traditional information processing and computing technologies have been difficult to effectively deal with the processing of big data. The effective processing of large-scale social network data faces major technical difficulties at multiple levels such as data storage and algorithm analysis. The huge amount of data makes it difficult for traditional stand-alone machine learning and data mining algorithms to complete calculations within an acceptable time, resulting in algorithm failure. Taking instan...

AI Technical Summary

Problems solved by technology

The huge amount of data makes it difficult for traditional stand-alone machine learning and data mining algorithms to complete calculations within an acceptable time, resulting in algorithm failure

Taking instant messaging data as an example, due to the characteristics of real-time update and variability, the dramatic increase in data volume will make the process of natural language processing and machine learning more complicated, and the overall architecture of the current parallel computing environment is not suitable for text Efficient parallel processing of data, especially process management and cache management, does not match the storage and distributed computing architecture required by text mining algorithms; moreover, the non-standard text increases the difficulty for users to understand information and discover events degree, which eventually leads to a serious imbalance in the data distribution

In addition, there is also the problem of lack of semantic representation of Chinese words, which makes traditional methods not fully applicable to the analysis and calculation of social texts, so it is difficult to realize information intelligence, human-computer interaction and automatic question answering

Images