A method for automatically learning Chinese construction based on correlation degree

Abstract

The application discloses a kind of Chinese construction automatic learning method based on correlation degree, which is divided into background statistical corpus, candidate construction generation corpus, MDL optimization corpus and determine the best correlation degree threshold corpus by corpus;Then the correlation degree of each information pair is calculated in the background statistical corpus, and the locally optimal correlation degree is searched on the small corpus for determining the best correlation degree threshold, so that the construction set obtained with the threshold has the best MDL evaluation index;Then the candidate construction set is generated using the locally optimal correlation degree on the larger corpus generated by candidate construction, and the correlation degree characteristics of candidate construction are calculated;Finally, the best MDL candidate construction set is searched by tabu search algorithm and variable neighborhood search algorithm on the MDL optimization corpus, and the Chinese construction set is obtained.The application can automatically learn to good quality Chinese construction, which can be directly applied to natural language processing technology, and has good application prospect.

Description

Technical Field

[0001] The present invention belongs to the field of Chinese construction learning, and particularly relates to an automatic Chinese construction learning method based on association degree. Background Art

[0002] The present invention solves the task of automatic Chinese construction learning. This task essentially searches for constructions in Chinese corpora, that is, searches for sequences of grammatical units that frequently co-occur in Chinese corpora, namely constructions. For example, the constructions with higher frequencies in the corpus are "n + ba + n + v" and "<positive comparison> + de + n", etc., and the constructions with lower frequencies are "a + dao + baobiao" and "xiao + n + v + zhe", etc. It is required to learn all the constructions existing in the Chinese corpus from the corpus.

[0003] In the technology of automatic construction learning, there is still a lack of automatic Chinese construction learning; currently, there is only a manually established modern Chinese construction library, and the number of constructions in this Chinese construction library is small, and it is necessary to manually identify and supplement constructions, with a high cost. Summary of the Invention

[0004] The purpose of the present invention is to provide an automatic Chinese construction learning method based on association degree in view of the deficiencies of the prior art.

[0005] The purpose of the present invention is achieved through the following technical solutions: an automatic Chinese construction learning method based on association degree, including the following steps:

[0006] (1) Represent the Chinese corpus as a word-segmented corpus with Chinese words as units, and represent the vocabulary, word nature, and semantic information of each word; divide the Chinese corpus into background statistical corpus, candidate construction generation corpus, MDL optimization corpus, and determine the best association degree threshold corpus;

[0007] (2) Use the background statistical corpus to complete the calculation of the association degree of each information pair; use the determine the best association degree threshold corpus to search for the locally optimal association degree threshold; use the candidate construction generation corpus and the locally optimal association degree threshold to generate a set of candidate constructions;

[0008] (3) Complete the following two-step search process in the MDL optimization corpus:

[0009] (3.1) Search for the best construction association degree feature combination through a tabu search algorithm to determine the candidate construction set with the best MDL,

[0010] (3.2) Search for the candidate construction set with the best MDL through a variable neighborhood search algorithm to obtain the result of automatically learned Chinese constructions.

[0011] Furthermore, the vocabulary and part-of-speech information in step (1) are obtained using the Chinese word segmentation methods jieba, LAC, PKUSEG, and HanLP.

[0012] Furthermore, the semantic information in step (1) is first obtained by using the CBoW model in Word2Vec to obtain static word vectors, and then the word vectors are clustered using the K-Means method. The clustering result is labeled as semantic information.

[0013] Furthermore, the correlation degree of each information pair in step (2) is a two-way correlation degree.

[0014] Furthermore, the method for searching the locally optimal correlation threshold in step (2) is a gridded search, which involves setting several correlation thresholds and selecting the correlation threshold with the smallest MDL value.

[0015] Furthermore, the method for generating the candidate construction set in step (2) is as follows: recursively search the vocabulary, part-of-speech and semantic information of the statement, with the recursion entry point being the terms of the statement and the recursion exit point being the terms with a relevance threshold less than the local optimum.

[0016] Further, the MDL calculation method in step (3) is as follows: the candidate construction set is regarded as the model, the MDL optimized corpus is regarded as the instance data, and the coding length of the candidate construction set and the coding length of the MDL optimized corpus are calculated using the minimum coding length formula. The sum of the two is the MDL value.

[0017] Furthermore, the features of the configuration correlation degree in step (3) include: average correlation degree, average correlation degree in the left and right directions, minimum correlation degree in the left and right directions, average correlation degree after removing the first term in the left and right directions, average correlation degree after removing the last term in the left and right directions, average correlation degree of the first and last terms in the left and right directions, correlation degree direction difference, and correlation degree direction strength.

[0018] Furthermore, the tabu search in step (3) is implemented by searching for the correlation combination of candidate configurations, and using Gaussian clustering to divide the candidate configurations into two groups, and selecting the candidate configuration set with the smaller MDL value as the latest candidate configuration set.

[0019] Furthermore, the variable neighborhood search in step (3) is implemented as follows: search for a subset of the latest candidate configuration set, and search the neighborhood by transforming it from small to large, and find the candidate configuration subset with the smallest MDL value. The candidate configuration subset is the configuration learning result.

[0020] Compared with the prior art, the beneficial effects of the present invention are:

[0021] (1) This invention can automatically learn high-quality Chinese constructions and has good application prospects. This invention solves the problems of existing manual construction databases, such as time-consuming and laborious manual updates, inconsistent word segmentation standards, and a small number of constructions that are difficult to directly apply to natural language processing technology.

[0022] (2) The method of the present invention processes the corpus through a word segmentation model, generates a large number of constructions from bidirectional correlation, and then learns the construction set with the minimum description length through a two-step search process, thus obtaining a large number of high-quality Chinese construction sets, which can be directly applied to subsequent natural language processing technologies.

[0023] (3) This invention can discover 9,152 high-quality Chinese constructions from a corpus of 5 million sentences, which is far more than the 1,000+ constructions in the current manually annotated Modern Chinese Construction Database. Its MDL compression rate is maintained at over 70%, which can reach the advanced level of current research. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structure learning process of the present invention;

[0025] Figure 2 This is a schematic diagram of the candidate configuration process for the correlation feature search of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0027] This invention addresses the unique characteristics of Chinese constructions by designing a multi-stage extraction and progressive refinement process for automatically acquiring Chinese construction sets, resulting in high-quality collections. To represent the sequence of grammatical units in Chinese, the corpus is segmented to obtain vocabulary and part-of-speech information. Then, static word vectors are trained, and semantic information is obtained through K-Means clustering. To address the lack of correlation between the frequency of constructions and grammatical unit sequences, a bidirectional correlation degree is used to describe the co-occurrence of constructions. Through techniques such as correlation degree calculation, correlation degree threshold learning, construction correlation degree features, the MDL (Multi-Level Definition) evaluation metric for construction sets, and heuristic search algorithms, the automatic learning of Chinese constructions is ultimately achieved.

[0028] See Figure 1 This invention provides an automatic Chinese structure learning method based on relevance, comprising the following steps:

[0029] (1) The Chinese corpus is represented as a word-segmented corpus, representing three layers of information for each word: vocabulary, part of speech, and semantics. The corpus is divided into background statistical corpus, candidate construction generation corpus, minimum description length (MDL) optimization corpus, and corpus for determining the optimal relevance threshold. The purpose of dividing the corpus into different corpora for construction learning is to learn more stable constructions. By calculating the relevance in the background statistical corpus, extracting the construction set in the candidate construction generation corpus, and screening candidate constructions in the MDL optimization corpus, the result of construction learning is not limited to one corpus, but is the result of the cooperation of multiple corpora, making the learned constructions more stable.

[0030] The vocabulary and part-of-speech information were obtained using Chinese word segmentation methods such as jieba, LAC, PKUSEG, and HanLP. The semantic information was first obtained by using the CBoW model in Word2Vec to obtain static word vectors, and then the word vectors were clustered using the K-Means method. The clustering results were labeled as semantic information.

[0031] (2) The correlation degree of each information pair is calculated using the background statistical corpus; the local optimal correlation degree threshold is searched using the corpus for determining the optimal correlation degree threshold; and a candidate construction set is generated using the candidate construction generation corpus and the local optimal correlation degree threshold.

[0032] The relevance of each information pair is bidirectional. The method for finding the locally optimal relevance threshold is a grid search, which involves setting several relevance thresholds and selecting the threshold that minimizes the MDL value of the generated construction. The candidate construction set is generated by recursively searching the lexical, part-of-speech, and semantic information of the statement. The recursive entry point is the item in the statement (containing three levels of information), and the recursive exit point is the item with a relevance threshold less than the locally optimal threshold.

[0033] (3) Complete the following two-step search process on the MDL optimized corpus:

[0034] (3.1) The best combination of configuration correlation features is searched by the tabu search algorithm (TS algorithm) to determine the candidate configuration set of the best MDL. The search granularity of step (3.1) is relatively large, which can filter out about half of the configurations. It belongs to coarse-grained search.

[0035] (3.2) Based on step (3.1), the candidate configuration set of the best MDL is searched by the variable neighborhood search algorithm (VNS algorithm), that is, the Chinese configuration result automatically learned. The search granularity of step (3.2) is small, which can filter out a small number of configurations, and belongs to fine-grained search.

[0036] The MDL calculation method is as follows: the candidate construction set is regarded as the model, the MDL optimized corpus is regarded as the instance data, and the minimum coding length formula is used to calculate the coding length of the candidate construction set and the coding length of the MDL optimized corpus. The sum of the two is the MDL.

[0037] The configuration correlation degree comprises 13 features, specifically: average correlation degree, average correlation degree in the left and right directions, minimum correlation degree in the left and right directions, average correlation degree after removing the first term in the left and right directions, average correlation degree after removing the last term in the left and right directions, average correlation degree of the first and last terms in the left and right directions, sum of correlation degree direction differences, and correlation degree direction strength. These correlation degree features of the configuration constitute the configuration correlation degree matrix P, providing search dimensions in coarse-grained search, enabling searching using configuration features rather than directly searching the configuration.

[0038] See Figure 2 The tabu search is implemented as follows: For the candidate configuration set C and its correlation matrix P, the tabu search algorithm (TS algorithm) searches for the sub-feature combination Q of the correlation degree, performs Gaussian clustering on Q to divide the candidate configuration set C into two, selects the candidate configuration set D with smaller MDL value, and performs the above process N times to obtain the candidate configuration set with the smallest MDL value, which is the latest candidate configuration set.

[0039] The variable neighborhood search is implemented as follows: search for a subset of the latest candidate configuration set, and search the neighborhood by transforming it from small to large to find the candidate configuration subset with the smallest MDL value. This candidate configuration subset is the configuration learning result.

[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0041] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for automatic learning of Chinese constructions based on relevance, characterized in that, Includes the following steps: (1) Represent the Chinese corpus as a word segment corpus based on Chinese words, and represent the lexical, part-of-speech and semantic information of each word; The Chinese corpus is divided into background statistical corpus, candidate construction generation corpus, MDL optimization corpus, and corpus for determining the optimal relevance threshold; (2) Use the background statistical corpus to complete the correlation calculation of each information pair; use the corpus for determining the optimal correlation threshold to search for the locally optimal correlation threshold; use the candidate construction generation corpus and the locally optimal correlation threshold to generate a candidate construction set; (3) Complete the following two-step search process in the MDL optimized corpus: (3.1) Search for the best combination of construction relevance features by tabu search algorithm to determine the candidate construction set of the best MDL. The MDL calculation method is as follows: regard the candidate construction set as the model and the MDL optimization corpus as the instance data. Use the minimum coding length formula to calculate the coding length of the candidate construction set and the coding length of the MDL optimization corpus. The sum of the two is the MDL value. The characteristics of structural relevance include: average relevance, average relevance in the left and right directions, minimum relevance in the left and right directions, average relevance after removing the first term in the left and right directions, average relevance after removing the last term in the left and right directions, average relevance of the first and last terms in the left and right directions, relevance direction difference, and relevance direction strength. The tabu search is implemented by searching for the correlation combination of candidate configurations, and using Gaussian clustering to divide the candidate configurations into two groups, selecting the set of candidate configurations with smaller MDL values ​​as the latest set of candidate configurations. (3.2) Search for the candidate configuration set of the best MDL by using the variable neighborhood search algorithm to obtain the automatically learned Chinese configuration result; the variable neighborhood search is implemented as follows: search for a subset of the latest candidate configuration set, and search the neighborhood by transforming from small to large, and search for the candidate configuration subset with the smallest MDL value. The candidate configuration subset is the configuration learning result.

2. The automatic Chinese structure learning method based on relevance as described in claim 1, characterized in that, The vocabulary and part-of-speech information in step (1) are obtained using the Chinese word segmentation methods jieba, LAC, PKUSEG, and HanLP.

3. The automatic Chinese structure learning method based on relevance as described in claim 1, characterized in that, The semantic information in step (1) is first obtained by using the CBoW model in Word2Vec to obtain static word vectors, and then by using the K-Means method to cluster the word vectors. The clustering result is labeled as semantic information.

4. The automatic Chinese structure learning method based on relevance as described in claim 1, characterized in that, The correlation degree of each information pair in step (2) is a two-way correlation degree.

5. The automatic Chinese structure learning method based on relevance as described in claim 1, characterized in that, The method for searching the local optimal correlation threshold in step (2) is a gridded search, which involves setting several correlation thresholds and selecting the correlation threshold with the smallest MDL value.

6. The automatic Chinese structure learning method based on relevance as described in claim 1, characterized in that, The method for generating the candidate construction set in step (2) is as follows: recursively search the vocabulary, part of speech and semantic information of the statement, with the recursion entry point being the terms of the statement and the recursion exit point being the terms with a relevance threshold less than the local optimum.

Images