Word segmentation method, device, equipment and readable storage medium based on multi-level dictionary

Abstract

This application discloses a word segmentation method based on multi-level dictionaries. The method uses at least two dictionaries to assist the word segmentation model for word segmentation. When representing a character, it not only generates a conventional vector representation, but also generates the character in at least two The feature representation in a dictionary, and finally determine the word-forming label of the character according to the vector representation and feature representation. This method improves the word segmentation performance of the overall scheme by distinguishing the status and importance of different words, and improves the domain adaptability and word segmentation accuracy. In addition, the present application also provides a word segmentation device, device and readable storage medium based on a multi-level dictionary, the technical effect of which is corresponding to the technical effect of the above method.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a word segmentation method, device, device and readable storage medium based on a multi-level dictionary. Background technique [0002] Chinese word segmentation is a process of dividing an input sentence into word sequences. An additional dictionary is usually provided for the model to alleviate the problem of insufficient training data for manual annotation. However, the current word segmentation schemes all use single-level dictionaries, ignoring the problem that different words in the dictionary have different word-forming probabilities, and also ignoring the problem that the same string becomes a word in one field but not in another field, resulting in The word segmentation effect of the word segmentation model is poor. [0003] The word segmentation method based on a single-level dictionary also has the problem of little influence on the actual word segmentati...

AI Technical Summary

Problems solved by technology

[0005] The purpose of this application is to provide a word segmentation method, device, device and readable storage medium based on a multi-level dictionary, to solve the problem that current word segmentation models all use single-level dictionaries, resulting in poor word segmentation performance

Images