Keyword vectorization method based on topic semantic information and application thereof

Abstract

The invention discloses a keyword vectorization method based on topic semantic information and an application thereof. The keyword vectorization method specifically comprises the following steps: firstly, generating a vector with document semantic information for each document by utilizing a Sension-BERT model; dimension reduction is carried out on the generated document vector through a UMAP dimension reduction algorithm, and local semantic features are highlighted; then, HDBSCAN topic clustering is carried out on the document vectors after dimension reduction, and each document is classified into one or more topics; and finally, calculating a subject term frequency-inverse subject frequency (TTF-ITF) score of each keyword in the subject by using a relationship between the document and the subject, and merging the keyword and the subject term frequency-inverse subject frequency (TTF-ITF) score of each subject to generate a final keyword vector. According to the method, high-precision keyword vectorization of topic semantic information is realized, and the method can be applied to topic word extraction, text classification and document retrieval.

Description

technical field [0001] The invention relates to the fields of natural language processing, text mining and searchable encryption, in particular to a keyword vectorization method based on subject semantic information and its application. Background technique [0002] With the continuous development of Internet technology and the advent of the era of big data, the scale of data has become increasingly large. In the face of large-scale and various types of data, how to classify these data, obtain keywords that are strongly related to a certain category, and effectively use them to guide practical activities is particularly important. Document data contains a large number of keywords and cannot be directly used by computers. Keyword vectorization is an effective means to solve this problem. Keyword vectorization plays an important role in the effective use of document data. For example, in an information retrieval scenario, given a search keyword, the user's retrieval intention...

AI Technical Summary

Problems solved by technology

Even if the keyword vectorization is performed based on the word2vec vectorization method, there is a problem of lack of semantic information of the reflected keywords;

[0004] In practical application scenarios, it is often necessary to vectorize keywords, such as the extraction of subject words in document classification, and vectorization of search keywords in the field of information retrieval; the vectorization method of keywords in the current prior art is mainly LDA The topic model, through the LDA topic model, can convert keywords into topic vectors, which reflect the relationship between keywords and topics, but the vectorization method based on the LDA topic model also has the problem of lack of semantic information of keywords reflected

Images