Big data text mining processing system and method

Abstract

The invention provides a big data text mining processing system and method, and can be applied to Internet service platforms, including search engines, social networks, instant communication, news information sites, electronic commerce, recreation and entertainment applications and the like. Firstly, mass text big data related to the behaviors, including browsing, communication, sharing, searching, downloading and the like, of users is obtained, and preprocessing, including data cleaning, word segmentation, stop word removal and the like, is executed for the text big data; and in addition, onthe basis of a user behavior mechanism, various forms of text data, including keywords, long texts, short texts and the like, are aggregated into a text link, feature extraction on the basis of the dynamic distribution of weight is executed by aiming at the text link, and mining analysis is realized according to extracted text link features.

Description

technical field [0001] The invention relates to the field of big data information processing and analysis, in particular to a big data text mining processing system and method thereof. Background technique [0002] In recent years, with the development and progress of network communication and computer technology, the storage, transmission and computing capabilities of information systems have grown by leaps and bounds, making the "big data era" a reality. Text big data is a very important part of big data information. Text big data information is data information in the form of text that exists in large quantities on various information platforms such as search engines, social networks, instant messaging, news sites, e-commerce, and leisure and entertainment applications. Valuable regularity information is obtained from text scattered information through text big data mining. [0003] The so-called text big data mining is to take text big data as the object and use approp...

AI Technical Summary

Problems solved by technology

Existing text big data mining technology cannot focus on specific users and effectively extract and represent the kernels that specific users care about

[0005] The defects of the existing technology are manifested in the following aspects: First, it cannot adapt to the mining and analysis of diversified text forms for specific users. With the diversification of network platforms and services, the forms of text big data related to specific users are becoming more and more abundant, including key It also includes texts in discrete forms such as words and tags, as well as long texts in the form of entire articles such as papers, blogs, news reports, and website posts, as well as short texts consisting of a few sentences such as Weibo comments and messages in Moments. That is to say, the text big data related to a specific user is a collection of diverse forms of text data, and the existing text big data mining and analysis methods are difficult to realize unified and effective semantic feature mining for text data sets containing diverse forms. In particular, the existing text mining analysis methods are mainly suitable for long texts, and it is difficult to effectively mine short texts and keyword texts generated by users.

Secondly, the existing text mining analysis methods extract the distribution characteristics of representative words in the text, which often cannot match well and describe the characteristics of the user's attention and interest in the text. article, but its concerns and interests are not necessarily the representative content identified as the characteristics of the article, but may be some non-representative local details of the article, so text mining in isolation often deviates from the real interest of users. where the

Images