Word segmentation method and device, electronic equipment and storage medium

Abstract

The invention discloses a word segmentation method and device, electronic equipment and a storage medium. The word segmentation method comprises the steps: inputting a word segmentation word stock into a pre-stored baseline word segmentation model, and determining a preliminary word segmentation result of the word segmentation word stock based on the baseline word segmentation model; inputting thepreliminary word segmentation result into a pre-trained word segmentation model, and outputting a segmentation result of the preliminary word segmentation result based on the word segmentation model,the segmentation result comprising a segmentation unit, and the segmentation unit comprising a segmentation character and/or a segmentation character set; and combining the segmentation units according to a preset combination rule, and determining a final word segmentation result of the segmented word stock. According to the word segmentation method, the existing baseline word segmentation modelis not changed, and the convergence rate of the word segmentation model is ensured, and the word segmentation efficiency is improved, and the word segmentation result of the baseline word segmentationmodel is corrected, so that the accuracy of the word segmentation result is improved.

Description

technical field [0001] The present invention relates to the technical field of word segmentation processing, in particular to a word segmentation method, device, electronic equipment and storage medium. Background technique [0002] Word segmentation refers to dividing a sequence of language characters into individual words. Word segmentation technology is the basis of text mining. For a piece of input text, successful word segmentation can achieve the effect of automatic computer recognition of the meaning of words and sentences, and realize natural speech processing. [0003] Commonly used word segmentation models are generally statistical-based word segmentation models or dictionary-based word segmentation models. The generalization ability of these two word segmentation models is generally relatively poor. Even if the supervised word segmentation model based on statistics has a certain generalization ability, due to the small amount of manually labeled corpus, the word ...

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a word segmentation method, device, electronic equipment and storage medium to solve the problems of low accuracy and low efficiency of word segmentation results when using a word segmentation model obtained after retraining in the prior art

Images