Big data processing method based on machine learning

Abstract

The invention provides a big data processing method based on machine learning, which comprises the following steps: given a retrieval sentence, filtering the words in the initial retrieval by using auniversal stop word list, and reserving meaningful retrieval words; the semantic block model is used to represent the lexical semantic vector. On the basis of semantic vector, cosine similarity is used to find out the closest words from other words for each initial search term, which can be used as extended search terms. The corresponding extended search terms in the initial search are used to replace the original search terms, and the newly generated search term sequence is used as the extended search terms. According to the permutation and combination of the extended search terms, the extended search sentences with different expressions are obtained. The invention improves the parallel frame of the MAPRUDUCE and is better adapted to the needs of text data mining. Aiming at the irregularity of social text, semantic vectors are used to represent and analyze the text data effectively, which is suitable for the analysis and calculation of social text mining of various scales.

Description

technical field [0001] The invention relates to big data mining, in particular to a machine learning-based big data processing method. Background technique [0002] Big data, especially social network data, contains huge commercial and social value. Effective management and utilization of these data and the value mining of data will have a huge impact on enterprises and individuals. On the other hand, while big data brings new development opportunities, it also brings many technical challenges. Traditional information processing and computing technology has been difficult to effectively deal with the processing of big data. The effective processing of large-scale social network data faces major technical difficulties at multiple levels such as data storage and algorithm analysis. The huge amount of data makes it difficult for traditional stand-alone machine learning and data mining algorithms to complete calculations within an acceptable time, resulting in algorithm failur...

AI Technical Summary

Problems solved by technology

The huge amount of data makes it difficult for traditional stand-alone machine learning and data mining algorithms to complete the calculation within an acceptable time, resulting in the failure of the algorithm

Taking instant messaging data as an example, due to the characteristics of real-time update and variability, the dramatic increase in data volume will make the process of natural language processing and machine learning more complicated, and the overall architecture of the current parallel computing environment is not suitable for text Efficient parallel processing of data, especially process management and cache management, does not match the storage and distributed computing architecture required by text mining algorithms; moreover, the non-standard text increases the difficulty for users to understand information and discover events degree, which eventually leads to a serious imbalance in the data distribution

In addition, there is also the problem of lack of semantic representation of Chinese words, which makes traditional methods not fully applicable to the analysis and calculation of social texts, so it is difficult to realize information intelligence, human-computer interaction and automatic question answering

Images