Sentence segmentation model establishing method and device

Abstract

The invention discloses a sentence segmentation model establishing method and device. The method comprises the steps: carrying out the word segmentation of each obtained corpus sentence, and determining words contained in the corpus sentence; determining rare words in words contained in the corpus sentence, and performing segmentation processing on the rare words by utilizing a sub-word segmentation algorithm; inputting a word sequence formed by words obtained after word segmentation processing and segmentation processing into the deep learning model for sentence segmentation labeling; According to the original sentence segmentation identifier of each corpus sentence and the sentence segmentation annotation corresponding to the corpus sentence output by the deep learning model, performingdeep learning on the corpus sentence. Adjusting parameters of the deep learning model, and establishing a sentence segmentation model, so that sentences without sentence segmentation identifiers are subjected to sentence segmentation processing by utilizing the established sentence segmentation model, the sentences which cannot be seen by the user are very long sentences without sentence segmentation identifiers any more, the readability and the comprehensibility of the sentences are improved, and the user experience can be improved.

Description

technical field [0001] The present application relates to the technical field of natural language processing, and in particular to a method and device for establishing a sentence segmentation model. Background technique [0002] In recent years, with the rapid development of speech recognition technology, there are more and more application fields of speech recognition, such as sending voice messages, voice memos, and simultaneous interpretation. [0003] However, when the voice information is recognized to obtain the corresponding sentence (the meaning of the sentence is complete or incomplete), the sentence does not have a sentence break mark, which is not conducive to reading and understanding, and when the voice information is relatively long, the recognized sentence When there are many characters in , it will be very difficult for users to read. Therefore, relevant personnel began to study how to segment sentences without sentence segmentation marks, so as to improve us...

AI Technical Summary

Problems solved by technology

[0003] However, when the voice information is recognized to obtain the corresponding sentence (the meaning of the sentence is complete or incomplete), the sentence does not have a sentence break mark, which is not conducive to reading and understanding, and when the voice information is relatively long, the recognized sentence When there are many characters in , it will be very difficult for users to read. Therefore, relevant personnel began to study how to segment sentences without sentence segmentation marks, so as to improve user experience

Images