A Chinese word segmentation method based on depth learning and forgetting algorithm

Abstract

The invention discloses a Chinese word segmentation method based on depth learning and forgetting algorithm, which comprises the following steps: 1. scanning sentences word by word to obtain natural language; dividing the scanned natural language into word sequences by depth learning word segmentation method and inputting them into a first word library; 2. scanning sentences word by word to obtainnatural language; 3. 2. Scan sentence verbatim to obtain natural language. 3, the word sequence in the first thesaurus and the candidate words in the second thesaurus are fused to obtain the final word segmentation result. The fusing method is: if the word sequence in the second thesaurus corresponds to the word in depth learning, the word sequence is merged into the word; a single word in the second thesaurus, if it corresponds to a word in depth learning, is combined forward or backward to form a word. The word segmentation method of the invention can automatically detect domain knowledge,complete the function of finding new words in the unsupervised domain and improve the word segmentation effect by integrating the depth learning word segmentation method and the forgetting algorithm word segmentation method.

Description

technical field [0001] The invention relates to the technical field of word segmentation, in particular to a Chinese word segmentation method based on deep learning and forgetting algorithms. Background technique [0002] Chinese Word Segmentation refers to dividing a sequence of Chinese characters into individual words. Word segmentation is the process of recombining continuous word sequences into word sequences according to certain specifications. [0003] 1. Word segmentation method based on string matching [0004] This method is also called the mechanical word segmentation method. It matches the Chinese character string to be analyzed with an entry in a "sufficiently large" machine dictionary according to a certain strategy. If a certain string is found in the dictionary, the match is successful. (a word is recognized). According to different scanning directions, the string matching word segmentation method can be divided into forward matching and reverse matching; a...

AI Technical Summary

Problems solved by technology

The existing word segmentation algorithm is based on the thesaurus, words that do not appear in the thesaurus cannot complete word segmentation

Images