Text segmentation and topic annotation for document structuring

Abstract

The invention relates to a method, a computer program product and a computer system for structuring an unstructured text by making use of statistical models trained on annotated training data. Each section of text in which the text is segmented is further assigned to a topic which is associated to a set of labels. The statistical models for the segmentation of the text and for the assignment of a topic and its associated labels to a section of text explicitly accounts for: correlations between a section of text and a topic, a topic transition between sections, a topic position within the document and a (topic-dependent) section length. Hence structural information of the training data is exploited in order to perform segmentation and annotation of unknown text.

Description

technical field [0001] The present invention relates to the field of generating structured documents from unstructured text by dividing the unstructured text into sections and assigning each section a semantic topic. Background technique [0002] Dividing text into segments and assigning each segment a label representing the segment's content is a fundamental but common task for constructing text documents. By utilizing relevant tags or headings, it is easy to retrieve within a document passages of text that are clearly relevant to the reader. Based on the tags, readers can quickly and efficiently identify the content relevance of text segments. Unfortunately, there are a large number of text documents that provide only insufficient structure or no structure at all. [0003] The collection of information provided by unstructured or weakly structured documents requires extensive reading and / or detailed description searching, which is tiring and very time-consuming for the r...

AI Technical Summary

Problems solved by technology

[0009] The basic strategy of first assigning sentences to clusters and then determining segment boundaries from cluster changes has several disadvantages: the method cannot be extended to capture information on longer ranges such as correlations about more distant segments, since these only appear at the completion of After cluster allocation

At the same time, substructures within segments (e.g. typical start phrases) cannot be captured in a sentence-by-sentence cluster assignment

Furthermore, explicit models for typical segment lengths cannot be interpolated in this method

Images