Dialogue short text clustering method based on form and semantic similarity

Abstract

The invention discloses a dialogue short text clustering method based on form and semantic similarity. The form similarity adopts character string editing distance similarity, and the semantic similarity is based on HowNet and WordNet knowledge bases; weight values of the short text and words are introduced during the calculation of the short text similarity. The dialogue short text clustering method based on the form and semantic similarity solves the problems of certain irregular and input wrong noise information, synonyms and semantic gaps included in the dialogue short text to a certain extent, and consequently, relatively great improvement is realized in comparison with a word bag vector based clustering method.

Description

technical field [0001] The invention belongs to the technical field of short text clustering, and relates to a method for clustering short texts of dialogues based on the similarity of string edit distance and the semantic similarity of words. Background technique [0002] With the rapid development of mobile communication and mobile Internet, various human-machine intelligent dialogue systems have emerged, such as Siri, google now, Xiaoi robot, etc. Taking Xiaoi Robot as an example, the number of users has exceeded 100 million, and there are 10 billion dialogue visits every year and a large amount of valuable dialogue text data are generated. These data are important data sources for user interest mining and knowledge base improvement of intelligent dialogue systems. Clustering analysis on these dialogue text data can gather similar dialogue texts and form several important cluster centers, which can improve the efficiency of mining user interests and extracting knowledge t...

AI Technical Summary

Problems solved by technology

[0005] (2) The topic is single, and a short dialogue text usually only discusses one thing

[0007] (4) Synonyms, mixed use of upper and lower case letters, and input errors are prominent

For example, Sahami et al. enter short texts into search engines to obtain the most relevant text sets returned, and these text data are used as auxiliary data information for corresponding short texts. This method solves the information sparsity of short texts to a certain extent, but A large amount of external auxiliary data is required, which leads to great restrictions on application scenarios

Another commonly used method is to use the knowledge base to expand the feature representation of words. For example, Hu et al. use WordNet or Wikipedia knowledge base to solve the problem of feature information sparsity. This method can supplement feature information from the semantic level, but for colloquial, wrong Short texts with severe noise cannot be handled well

Images