Standard lexicon word segmentation method, device and equipment and computer readable storage medium

Abstract

The invention provides a standard word library word segmentation method, device and equipment and a computer readable storage medium. The method comprises: standard words in a standard word library tobe segmented are scattered into single Chinese characters to form an original Chinese character library, and a first adjacent probability and a first Bayesian probability between every two Chinese characters in the original Chinese character library are calculated; performing a Chinese character merging operation on the original Chinese character library according to the first adjacent probability and the first Bayesian probability to obtain a to-be-adjusted Chinese character library; judging whether the minimum adjacent probability in the second adjacent probability between every two Chinesecharacters in the to-be-adjusted Chinese character library is greater than a preset threshold value or not; if yes, according to a second adjacent probability and a second Bayesian probability, executing a Chinese character combination operation on the to-be-adjusted Chinese character library until the minimum adjacent probability in the adjacent probabilities between every two Chinese charactersin the obtained target Chinese character library is smaller than or equal to a preset threshold value; otherwise, outputting the combined Chinese character groups as standard words. According to themethod, the word segmentation accuracy of the standard lexicon and the universality of the standard lexicon are improved.

Description

technical field [0001] The present invention relates to the technical field of natural language processing, in particular to a standard thesaurus word segmentation method, device, equipment and computer-readable storage medium. Background technique [0002] NLP (Natural Language Processing, Natural Language Processing) is a sub-field of artificial intelligence. Currently, NLP is mainly calculated through the existing word segmentation database, which can be performed through dictionaries or artificially assisted annotation. [0003] When segmenting the standard thesaurus through a dictionary or manual assisted annotation, due to the limited dictionary or manual assisted annotation, it is impossible to accurately segment the standard thesaurus, making the standard thesaurus unsuitable for NLP applications in specific fields, such as the medical field , it is necessary to re-segment the existing standard thesaurus. [0004] Therefore, how to improve the word segmentation accu...

AI Technical Summary

Problems solved by technology

[0003] When segmenting the standard thesaurus through a dictionary or manual assisted annotation, due to the limited dictionary or manual assisted annotation, it is impossible to accurately segment the standard thesaurus, making the standard thesaurus unsuitable for NLP applications in specific fields, such as the medical field , it is necessary to re-segment the existing standard thesaurus

Images