A text aggregation method and system

Abstract

The present invention provides a text aggregation method and system for aggregating long texts and short texts. The method includes the steps of: clustering long texts to obtain topics corresponding to long texts, and the long texts include Title; establish a classification model, and obtain the abstract and entity collection of long text; use the topic, title, abstract and entity collection to establish the first mapping set and the second mapping set; use the first mapping set and the second mapping Set training described classification model, obtain the classification model after training; Obtain the abstract of long text to be measured, utilize the abstract of described long text to be measured and the short text to be measured to establish the 3rd mapping set, utilize described 3rd mapping set and The trained classification model obtains text aggregation results; the method and system use the entity features of long texts and short texts to filter out short texts that contain the same entities as long texts, reducing computational complexity and ensuring text aggregation efficiency.

Description

technical field [0001] The present invention relates to the technical field of text clustering, and more specifically, to a text aggregation method and system. Background technique [0002] There are many sources of information in real life, including professional media websites, self-media platforms, and social media. It has become a development trend to integrate multiple semantically related information and short comments. For example, in scenarios such as displaying search results and presenting news information, most of them are displayed in the form of topics rather than single texts, which can integrate multiple news sources, reduce information redundancy, and provide users with richer information. [0003] In the process of integrating information and short comments, that is, in the process of integrating long texts and short texts, clustering algorithms are generally required. Clustering algorithms can aggregate and generate topics contained in news. Clustering alg...

AI Technical Summary

Problems solved by technology

Clustering algorithms can aggregate and generate topics contained in news. Clustering algorithms are usually only used in long texts, and the effect is usually not ideal when used in short texts, because long texts have relatively more vocabulary, which can provide a stable and rich Long text features are expressed; while short texts have less vocabulary and lack of context information, making it difficult to form valuable clusters

For short text integration, the general method is to use classification algorithms, but the classification algorithm requires manual preset topic types, but due to the multi-dimensionality of natural language data, it is easy to have the problem that short text content does not exactly match the preset types

Images