Big data processing method based on machine learning

Abstract

The invention provides a big data processing method based on machine learning, comprising: given a search sentence, using a common stop word list to filter the words in the initial search, and retaining meaningful search terms; using a semantic block model Represent vocabulary with semantic vectors; on the basis of semantic vectors, use cosine similarity for each initial search term to find multiple words with the closest similarity from other vocabulary as extended search terms; use the corresponding The extended search terms in the initial search sentence are replaced, and the newly generated search word sequence is used as the extended search sentence; according to the arrangement and combination of the extended search words, the extended search sentences with different expression forms are obtained. The invention improves the parallel framework of MAPRUDUCE to better meet the needs of text data mining; and aims at the non-standard characteristics of social text, uses semantic vectors to effectively represent and analyze text data, and is suitable for social text mining analysis of various scales and calculate.

Description

technical field [0001] The invention relates to big data mining, in particular to a machine learning-based big data processing method. Background technique [0002] Big data, especially social network data, contains huge commercial and social value. Effective management and utilization of these data and the value mining of data will have a huge impact on enterprises and individuals. On the other hand, while big data brings new development opportunities, it also brings many technical challenges. Traditional information processing and computing technologies have been difficult to effectively deal with the processing of big data. The effective processing of large-scale social network data faces major technical difficulties at multiple levels such as data storage and algorithm analysis. The huge amount of data makes it difficult for traditional stand-alone machine learning and data mining algorithms to complete calculations within an acceptable time, resulting in algorithm fai...

AI Technical Summary

Problems solved by technology

The huge amount of data makes it difficult for traditional stand-alone machine learning and data mining algorithms to complete calculations within an acceptable time, resulting in algorithm failure

Taking instant messaging data as an example, due to the characteristics of real-time update and variability, the dramatic increase in data volume will make the process of natural language processing and machine learning more complicated, and the overall architecture of the current parallel computing environment is not suitable for text Efficient parallel processing of data, especially process management and cache management, does not match the storage and distributed computing architecture required by text mining algorithms; moreover, the non-standard text increases the difficulty for users to understand information and discover events degree, which eventually leads to a serious imbalance in the data distribution

In addition, there is also the problem of lack of semantic representation of Chinese words, which makes traditional methods not fully applicable to the analysis and calculation of social texts, so it is difficult to realize information intelligence, human-computer interaction and automatic question answering

Images