Paragraph segmentation method and paragraph segmentation device

Abstract

PROBLEM TO BE SOLVED: To solve such a problem that in the conventional method, it is difficult to correctly divide a passage when a plurality of passages containing sentences with kindred meaning and similar feature quantity are included in one document.SOLUTION: A passage division device 100, under control of a control unit 101, divides a document input from an input unit 102 into sentence units at a sentence division unit 103. A feature quantity calculation unit 104, with the divided sentence as a query, performs associative retrieval of a document which is stored beforehand in a corpus unit 111 and acquires a document vector. A similarity calculation unit 105 retrieves two document vectors whose similarity becomes maximum, and when the similarity is equal to or larger than a prescribed threshold, a retrieval query generation unit 106 consolidates the two sentences to generate a query as a common element. The feature quantity calculation unit 104 regenerates a document vector by using this query. A feature quantity update unit 107 updates the feature quantity on the basis of its reliability, and connects corresponding sentences sequentially to make a passage while updating the feature quantity.

Description

technical field [0001] The invention relates to the processing of electronic files, in particular to the paragraph segmentation technology of electronic files. Background technique [0002] In recent years, digitization and databaseization of documents have been advanced, and natural language processing technology has also made great progress. For example, a large number of researches have been carried out for automatic summarization of documents and automatic keyword extraction for document retrieval. However, in many cases, a document that is a target of these techniques is assumed to be a document that is divided into each paragraph, that is, each topic or content summary unit of meaning, or includes only a single paragraph. Therefore, for documents containing multiple paragraphs, it is effective to pre-split paragraphs. Conventionally, as such a paragraph segmentation method, a text segmentation method described in Patent Document 1 or Patent Document 2 or the like is k...

AI Technical Summary

Problems solved by technology

[0003] However, in the existing methods related to paragraph segmentation and text segmentation, it is difficult to correctly segment paragraphs when a document contains multiple paragraphs containing sentences with similar meanings, that is, sentences with similar feature quantities.

As a result, automatic summarization of documents or automatic keyword extraction for document retrieval cannot be performed efficiently

Images