A text segmentation method based on a layered Dirichlet model

Abstract

The invention belongs to the technical field of text segmentation, and particularly relates to a text segmentation method based on a layered Dirichlet model, which comprises the following steps: S1, acquiring a news corpus, preprocessing the news corpus to obtain a word segmentation set of the whole news corpus, and performing word frequency statistics on the word segmentation set; S2, putting a result after word frequency statistics into a layered Dirichlet process model for training, and storing the trained layered Dirichlet process model; And S3, obtaining a topic vector of each word in theto-be-segmented text through the trained layered Dirichlet process model, and realizing text segmentation according to the topic vectors. By using the method and the device, the following effects canbe realized: the method enables text segmentation not to depend on manual setting of the number of topics, the topic vectors are automatically generated through the hierarchical Dirichlet process model, and the text segmentation efficiency is improved.

Description

technical field [0001] The invention belongs to the technical field of text segmentation, and in particular relates to a text segmentation method based on a hierarchical Dirichlet model. Background technique [0002] With the rapid development of the network, people have gradually stepped into a new network era, and various electronic text information is growing at an explosive rate. While all kinds of massive information bring convenience to society, it also brings huge challenges to text processing and analysis, such as how to quickly and accurately obtain effective information from this massive information. Text segmentation is to segment the text based on the principle of topic correlation, so that each semantic paragraph has the minimum similarity, and each semantic paragraph has the maximum similarity, so as to find the boundaries of different topics. [0003] Commonly used methods for text segmentation include methods based on vocabulary aggregation, methods based on...

AI Technical Summary

Problems solved by technology

In practical applications, the setting of the number of topics has a great influence on the text segmentation effect. If the number of topics is set too high, it will cause training overfitting, and if it is set too low, the description of the text will not be comprehensive enough.

[0004] Traditional text segmentation algorithms generally rely on the manual setting of the number of topics. It is difficult to estimate the number of topics in a large corpus, which may easily cause overfitting or incomplete description of the text.

Images