Semantic clustering method

Abstract

The invention belongs to the technical field of data mining, particularly relates to a semantic clustering method, and provides the following scheme that the semantic clustering method specifically comprises the following steps of S1, data preprocessing; s2, performing deep network characterization statistical characterization; s3, carrying out word segmentation representation; s4, calculating the similarity; s5, converting into an adjacent matrix; s6, constructing a graph; s7, calculating a connected domain; and S8, result clustering: in the step S1, many information which interferes with or is useless to semantics exists in dialogue data, such as noise of Emoji expressions, special custom symbols, websites, dialogue references and the like, the noise is cleaned through regular expressions, and dialogues with short dialogue data length and insufficient semantics after cleaning are filtered out. According to the method, on the basis of combined representation of deep learning and a TFIDF statistical method, similarity of the combined representation is converted into a graph adjacency matrix to construct a graph, and the clustering purpose is achieved by calculating a connected domain of the graph.

Description

technical field [0001] The invention relates to the technical field of data mining, in particular to a semantic clustering method. Background technique [0002] Clustering is a commonly used analysis method in machine learning and data mining. The traditional clustering idea is to extract features from data, use these features to represent corresponding data, and then quantify the similarity and correlation between features according to metrics. characteristics, and group the same and similar features into one category, so as to achieve the purpose of data clustering. Commonly used metrics include Euclidean distance, cosine similarity, etc. Commonly used features for semantic clustering include TFIDF, TopicModel, and text representation based on deep learning, combined with clustering algorithms to achieve the goal. Commonly used clustering algorithms include KMeans, hierarchical clustering, spectral clustering, etc., but these methods have the following shortcomings: [...

AI Technical Summary

Problems solved by technology

[0003] (1) Some clustering methods, such as KMeans, need to set the number of cluster categories in advance. If there is no prior, it is difficult to determine an appropriate number of clusters

Although there are some theories to guide the determination of the number of clusters, it is still weak and consumes more computing resources when more accurate clustering is required.

[0004] (2) For other clustering methods, although the number of clustering categories does not need to be determined in advance, it requires a lot of calculations and some hyperparameter settings, and the clustering results are often unsatisfactory, such as hierarchical clustering

[0005] (3) In addition, there are some clustering methods, the concept is obscure to non-professionals, when interaction clustering is required, it is difficult to explain the meaning of interaction parameters, such as spectral clustering

Images