News sentence clustering method based on semantic similarity, device and storage medium

Abstract

The invention provides a news sentence clustering method based on semantic similarity. The method includes the following steps: preprocessing news sentences of a corpus, and extracting available words; utilizing the available words to train a continuous bag-of-words model to obtain an initial word vector of each available word; utilizing an initial sentence vector of each news sentence and the initial word vectors of the left and right adjoining available words of a certain available word in the news sentence to train the continuous bag-of-words model in an iterative manner to obtain a currentword vector of each available word in the news sentence and a final sentence vector of the news sentence; merging an average value of the word vectors of all the available words, one-hot vectors of high-frequency words and the final sentence vector of each news sentence to obtain a semantic vector of the news sentence; and calculating distances between the semantic vectors to obtain the semanticsimilarity between the different news sentences, and clustering the news sentences of the corpus in accordance therewith. The invention also provides an electronic device and a computer-readable storage medium.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a news sentence clustering method, device and storage medium based on semantic similarity. Background technique [0002] With the development of Internet technology, the amount of information owned by humans has shown explosive growth, and the amount of news text data has also increased rapidly. In the face of a huge amount of news corpus, clustering sentences can facilitate the inductive analysis of similar news and realize the comprehensive utilization of news data. [0003] The current commonly used news clustering methods often ignore the importance of word elements when calculating and comparing sentence similarities. The existence of a large number of polysemous words makes it easy for these clustering methods to gather different types of information together. There are also some news clustering methods that take into account the role of semantic information behind the s...

AI Technical Summary

Problems solved by technology

[0003] The current commonly used news clustering methods often ignore the importance of word elements when calculating and comparing sentence similarity. The existence of a large number of polysemous words makes it easy for these clustering methods to gather different types of information together.

There are also some news clustering methods that take into account the role of semantic information behind the surface text information of keywords, and use the knowledge platform with rich semantic concepts as an intermediate reference space to calculate the similarity of news sentences, but it is difficult to overcome the gap between clustering quality and efficiency. contradiction

Images