Word segmentation method and device, terminal and computer readable storage medium

Abstract

The embodiment of the invention provides a word segmentation method and device, a terminal and a computer readable storage medium. The method comprises the following steps: determining text information of a to-be-segmented word; performing word segmentation on the text information according to a preset matching algorithm, a unitary word bank in a pre-constructed word segmentation model and a wordindex table corresponding to the unitary word bank; wherein the words in the unitary lexicon are sorted according to the number of characters contained in each word; wherein the word index table is used for indexing the position of each character number of words in the unitary word bank; And obtaining a word segmentation result of the text information. Therefore, in the process of carrying out word segmentation on the text information by utilizing the preset matching algorithm, the position interval corresponding to the character number of the word to be queried can be determined by utilizingthe word index table, and then whether the word exists in the position interval or not can be searched. In this way, traversing of the unitary word bank is avoided, the searching time is shortened, and therefore the word segmentation speed is increased.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a word segmentation method, device, terminal and computer-readable storage medium. Background technique [0002] Since Chinese uses characters as the writing unit, and there is usually no obvious participle mark between characters in a sentence. Therefore, electronic devices often need to segment the sequence of Chinese characters into individual words, so as to understand the semantics expressed by the sequence of Chinese characters according to the obtained word segmentation results. [0003] Currently, commonly used word segmentation algorithms include dictionary-based word segmentation algorithms. Wherein, the word segmentation algorithm is to match the Chinese character string to be matched with each word in a preset dictionary containing a large number of phrases according to a certain matching algorithm. If a word can be found in the dictionary, it means ...

AI Technical Summary

Problems solved by technology

[0004] The inventor found in the process of realizing the present invention that the word segmentation speed of this word segmentation method is still relatively slow and cannot meet the needs of fast word segmentation

Images