Chinese word segmentation method based on dynamic programming

Abstract

The invention discloses a Chinese word segmentation method based on dynamic programming, and relates to the technical field of natural language processing, and the method comprises the steps: S1, obtaining at least one piece of Chinese text information, and analyzing the basic attributes of the Chinese text information; and S2, collecting all words to construct a basic word bank, dividing common names of all fields, dividing common words into high-frequency words to improve word frequency weights, selecting related fields according to needs before word segmentation work, and improving special word weights of the related fields. According to the method, a dynamic planning method is adopted to consider all conditions which possibly form words in sentences, the weight of a word segmentation scheme is determined by the frequency of the corresponding words in a word list, the optimal word segmentation scheme is determined according to the word frequency, the word frequency can be updated and new words can be supplemented according to a training result, and therefore a special word bank suitable for a self-applicable scene is constructed; therefore, the subsequent Chinese word segmentation work is optimized, and the method has better applicability and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a Chinese word segmentation method based on dynamic programming. Background Art

[0002] Dynamic programming algorithms are often used to solve problems with some optimal properties. In such problems, there may be many feasible solutions. Each solution corresponds to a value, and we hope to find the solution with the optimal value. The dynamic programming algorithm is similar to the divide-and-conquer method. Its basic idea is to decompose the problem to be solved into several sub-problems, solve the sub-problems first, and then get the solution to the original problem from the solutions to these sub-problems.

[0003] Chinese word segmentation is the most critical preprocessing in Chinese text information processing and the basis of text mining. Chinese word segmentation is the basis of other Chinese information processing, such as Chinese search engines, machine translation, speech synthesis, automatic classification, automatic summarization, automatic proofreading, etc., all of which require word segmentation.

[0004] When processing Chinese text objects, a very critical issue lies in the word segmentation link. Almost all subsequent results depend on the first step of word segmentation. Therefore, the accuracy of word segmentation greatly affects the subsequent processing. Different word segmentation results also affect the extraction of features. This paper proposes a Chinese word segmentation method based on dynamic programming. The dynamic programming method is introduced to list the various possibilities of Chinese word segmentation according to probability, and the most frequent one in the word list is used as the best word segmentation scheme. The words in the scheme results are recorded to increase the word frequency. According to the word segmentation results, the unregistered new words can be recorded and supplemented, thereby constructing a special word library for its own applicable scenarios. Summary of the invention

[0005] The present invention provides a Chinese word segmentation method based on dynamic programming. The dynamic programming method is used to consider all possible word formation situations in a sentence. The weight of the word segmentation scheme is determined by the frequency of the corresponding word in the vocabulary. The best word segmentation scheme is determined according to the word frequency, and the word frequency can be updated and new words can be added according to the training results, so as to construct a special vocabulary suitable for its own applicable scenarios, thereby optimizing the subsequent Chinese word segmentation work.

[0006] The object of the present invention is to provide a Chinese word segmentation method based on dynamic programming, and the method comprises the following steps:

[0007] Step S1, obtaining at least one Chinese text information and analyzing basic attributes of the Chinese text information;

[0008] Step S2, collect all words to build a basic word library, divide the common names in various fields, and divide the common words into high-frequency words to increase the word frequency weight. Before the word segmentation work, select the relevant fields as needed to increase the weight of the special words in the relevant fields;

[0009] Step S3, performing word group prediction on all characters in the text information according to the word library to obtain multiple word group results;

[0010] Step S4, only count the high-frequency words in the multiple word segmentation results, obtain the weight information of the high-frequency words, and compare the multiple groups of word segmentation structures. The one with the highest sum of comparison weights is the optimal word segmentation result, and the word frequency weights from low to high are the subsequent supplementary word segmentation structures. The optimal word segmentation result is output as the predicted word segmentation structure;

[0011] Step S5, training and judging the best predicted word segmentation result, outputting the Chinese word segmentation result and recording the word segmentation result if it is correct, and improving the frequency of the word segmentation words in the basic word library;

[0012] Step S6, if the best prediction segmentation result is incorrect, the training is judged in order from high to low according to the word frequency weights until it is correct or the judgment is completed;

[0013] Step S7: If all word segmentation structures are incorrect, new words that have not been registered are manually added into the basic word library and the word segmentation process is performed again.

[0014] Furthermore, the basic attributes of the Chinese text information in S1 include characters and the number of characters.

[0015] Furthermore, the frequency weights of specialized words in the relevant field in S2 are higher than the frequency weights of common words.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. Collect all words to build a basic vocabulary, divide the common names in various fields, and divide the common words into high-frequency words to increase the word frequency weight. Before the word segmentation work, select relevant fields according to needs and increase the weight of special words in related fields. The word frequency weight of special words in related fields is higher than the word frequency weight of common words. Use dynamic programming method to consider all possible word formation situations in the sentence. The weight of the word segmentation scheme is determined by the frequency of the corresponding word in the vocabulary, and the best word segmentation scheme is determined according to the word frequency;

[0018] 2. The word frequency can be updated and new words can be added according to the training results, so as to build a special word library suitable for its own applicable scenarios, and then optimize the subsequent Chinese word segmentation work with better applicability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 A system flow chart of a Chinese word segmentation method based on dynamic programming provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following is combined with Figure 1 The principles and features of the present invention are described, and the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, the present invention provides a technical solution:

[0023] The object of the present invention is to provide a Chinese word segmentation method based on dynamic programming. In step S1, at least one Chinese text information is obtained, and basic attributes of the Chinese text information are analyzed. The basic attributes of the Chinese text information include characters and the number of characters.

[0024] Step S2, collect all words to build a basic word library, divide the common names in various fields, and divide the common words into high-frequency words to increase the word frequency weight. Before the word segmentation work, select the relevant fields as needed, increase the weight of the special words in the relevant fields, and the word frequency weight of the special words in the relevant fields is higher than the word frequency weight of the common words;

[0025] Step S3, performing word group prediction on all characters in the text information according to the word library to obtain multiple word group results;

[0026] Step S4, only count the high-frequency words in the multiple word segmentation results, obtain the weight information of the high-frequency words, and compare the multiple groups of word segmentation structures. The one with the highest sum of comparison weights is the optimal word segmentation result, and the word frequency weights from low to high are the subsequent supplementary word segmentation structures. The optimal word segmentation result is output as the predicted word segmentation structure;

[0027] Step S5, training and judging the best predicted word segmentation result, outputting the Chinese word segmentation result and recording the word segmentation result if it is correct, and improving the frequency of the word segmentation words in the basic word library;

[0028] Step S6, if the best prediction segmentation result is incorrect, the training is judged in order from high to low according to the word frequency weights until it is correct or the judgment is completed;

[0029] Step S7: If all word segmentation structures are incorrect, new words that have not been registered are manually added into the basic word library and the word segmentation process is performed again.

[0030] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0031] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A Chinese word segmentation method based on dynamic programming, characterized in that: The method is as follows: Step S1, obtaining at least one Chinese text information and analyzing basic attributes of the Chinese text information; Step S2, collect all words to build a basic word library, divide the common names in various fields, and divide the common words into high-frequency words to increase the word frequency weight. Before the word segmentation work, select the relevant fields as needed to increase the weight of the special words in the relevant fields; Step S3, performing word group prediction on all characters in the text information according to the word library to obtain multiple word group results; Step S4, only count the high-frequency words in the multiple word segmentation results, obtain the weight information of the high-frequency words, and compare the multiple groups of word segmentation structures. The one with the highest sum of comparison weights is the optimal word segmentation result, and the word frequency weights from low to high are the subsequent supplementary word segmentation structures. The optimal word segmentation result is output as the predicted word segmentation structure; Step S5, training and judging the best predicted word segmentation result, outputting the Chinese word segmentation result and recording the word segmentation result if it is correct, and improving the frequency of the word segmentation words in the basic word library; Step S6, if the best prediction segmentation result is incorrect, the training is judged in order from high to low according to the word frequency weights until it is correct or the judgment is completed; Step S7: If all word segmentation structures are incorrect, new words that have not been registered are manually added into the basic word library and the word segmentation process is performed again.

2. The Chinese word segmentation method based on dynamic programming according to claim 1, characterized in that: The basic attributes of the Chinese text information in S1 include characters and the number of characters.

3. The Chinese word segmentation method based on dynamic programming according to claim 1, characterized in that: The frequency weights of specialized words in the relevant field in S2 are higher than the frequency weights of common words.

Images