Novel semantic analysis method and system for large-scale document themes

Abstract

The invention discloses a novel semantic analysis method and system for large-scale document themes. The method comprises the steps of detecting whether classification information exists in a known document set or not, and if yes, executing a supervised subject generation step to generate and obtain a plurality of theme sets; otherwise, executing an unsupervised subject generation step to generate and obtain a plurality of theme sets; and according to the obtained theme sets, calculating a degree of correlation between a document needed to be analyzed and each theme set, thereby obtaining a theme distribution state, about the theme set, of the document. The system comprises a theme set generation unit and a theme analysis unit. According to the method and the system, theme generation can be automatically, quickly, flexibly and effectively finished in large-scale document data, and theme distribution of any given document in theme generation can be analyzed and assessed. The method and the system are suitable for an occasion of quickly generating the themes.

Description

technical field [0001] The present invention relates to the technical field of semantic analysis, in particular to a novel large-scale document topic semantic analysis method and system. Background technique [0002] In the era of big data, the number of documents has grown at an unprecedented rate, exceeding the time and energy costs of manual processing. A large amount of data accumulated in daily life: from text files to office files, as well as data in the form of documents such as pictures, images, video and audio, etc. are often not fully utilized, and there are a large number of excavable and learning information. Regarding the huge information value hidden in the data, the reasons why people have no way to start are, firstly, the diversity of document types and document sources, and secondly, the high latitude and unstructured characteristics of the document content. The key is the large amount of document data. At present, big data analysis, especially the subject...

AI Technical Summary

Problems solved by technology

However, such dictionaries cannot cover all possible semantic situations, especially for vocabulary and domain-specific knowledge that have not appeared in dictionaries.

The LDA topic model obtains the semantic relationship between documents, topics, and words by calculating the statistical information of the corpus. However, because it continues to use the bag of words model, it cannot avoid the disaster of dimensionality caused by a large vocabulary. Furthermore, the iteration of the LDA model training Matrix operations lead to too long topic training time

Images