Self-learning word segmentation method and device, computer equipment and storage medium

Abstract

The invention discloses a self-learning word segmentation method. The method comprises the following steps of obtaining missed search hot words; identifying whether the missed search hot words need tobe segmented or not; if word segmentation needs to be carried out, adding the missed search hot words to a remote word segmentation dictionary; periodically scanning the remote word segmentation dictionary, and judging whether the change amount of the remote word segmentation dictionary meets index reconstruction conditions or not; if a re-indexing condition is met, re-establishing a search indexby a sub-node; and after the sub-node reestablishes the search index, resetting the search hot word lexicon in the sub-node. The invention further discloses a device for realizing the self-learning word segmentation method, computer equipment and a storage medium. The hot words are selected in a real-time self-learning mode, the remote word segmentation dictionary is updated, uninterrupted updating of ElasticSearch search services is achieved, the search accuracy is effectively improved, and the word segmentation accuracy is improved by optimizing the dictionary.

Description

technical field [0001] The present invention relates to the field of computer technology, and in particular, to a self-learning word segmentation method, device, computer equipment and storage medium. Background technique [0002] ElasticSearch is a Lucene-based search server that provides a full-text search engine with distributed multi-user capabilities. The word segmentation task is mainly used in the process of indexing by ElasticSearch. The words passed through the word segmentation system can be recognized by the search engine and returned in the next search process. However, some words cannot be successfully segmented by the word segmentation system, or there are some hot words. The generation of , also cannot be processed by the word segmentation system. At this time, the mainstream practice is to update the word segmentation dictionary and re-establish the search index. However, this method has the following drawbacks: [0003] 1. The update speed is slow, and th...

AI Technical Summary

Problems solved by technology

[0003] 1. The update speed is slow, and users need to rebuild the index every time they update, which consumes a lot of computing resources;

[0004] 2. It is impossible to update in time first, and at the same time, the new words are not screened, which may cause some words that do not need word segmentation to enter the word segmentation dictionary, polluting the index

Images