A new word discovery method combining word vector multi-feature fusion

Abstract

The present invention discloses a new word discovery method that combines word vector multi-feature fusion. The method comprises the following steps: feature labeling of a corpus, calculating a mutual information matrix and a self-information matrix for attribute pairs, and calculating scores for candidate feature attribute pairs; clustering word vectors, finding the corresponding nearest neighbor class center vector, and finding the word corresponding to the class center vector as the word vector feature attribute, traversing all word vector feature attribute pairs, calculating the self-information matrix and the mutual information matrix for the word vector feature attribute pairs, and calculating the scores of all word vector feature attribute pairs; configuring optimized weight coefficients for the scores of all candidate feature attribute pairs and summing them to calculate a total score; extracting the corresponding score vectors of the candidate word pairs to a sample set when the total score exceeds a new word threshold; updating the weight coefficients and iteratively training until the loss function converges to obtain the final weight coefficients; recalculating the total scores of the candidate word pairs, and outputting new words that exceed the new word threshold. The present invention realizes a more reasonable new word evaluation method and more effectively mines potential new words.

Description

Technical Field

[0001] The present invention relates to the technical field of new word mining and classification, and in particular to a new word discovery method combining word vector multi-feature fusion. Background Art

[0002] With the rapid development of the internet, new words (especially internet slang) are constantly emerging. Unlike Latin-based languages, Chinese, Japanese, Korean, and other languages ​​lack specific word separators. Therefore, downstream tasks (such as text classification, information extraction, semantic search, and machine translation) require word segmentation. Otherwise, a large amount of entity and part-of-speech information can be lost, resulting in reduced performance. Unknown words (also known as new words) have the greatest impact on word segmentation.

[0003] In the existing technology, the most commonly used unsupervised new word discovery method is based on self-information and mutual information. It uses two aspects to measure whether two subwords (characters) should form a new word: 1. Mutual information: the degree of co-occurrence of the two subwords (characters); 2. Self-information: the richness of the left and right words of the two subwords (characters);

[0004] In addition to the above basic new word discovery algorithms based on word (character) segmentation features, relevant natural language researchers have proposed improved methods based on multi-feature fusion such as part of speech or dependency syntax analysis;

[0005] With the development of deep learning and neural networks, word embeddings, as the most fundamental elements of natural language processing, have garnered increasing attention. As intermediate products trained on large-scale corpora, word embeddings offer advantages over one-hot embeddings, such as lower dimensionality and greater grammatical and semantic information. Existing techniques also focus on word embedding features, calculating frequent n-gram patterns using similarity scores between word embeddings and further filtering candidate new words through the addition of pruning algorithms. Other approaches have proposed recognition methods that combine conditional random fields (CRFs) with word embeddings.

[0006] The aforementioned existing technologies all share the common approach of starting with a single (or two) simple features, or deeply exploring other potential factors within that feature that could improve mining results (e.g., using the TextRank algorithm to obtain a seed dictionary or post-pruning algorithms). These approaches fail to comprehensively integrate all features, such as parts of speech and syntactic dependencies. Many natural language processing tools even offer other features like named entity recognition. Furthermore, existing technologies are limited to a single feature, and the limitations of that feature often determine the ceiling of new word discovery accuracy. Various optimization methods and parameter adjustment approaches can only approximate this theoretical ceiling.

[0007] Secondly, the currently selected features, such as part of speech, differ significantly from the granularity of word segmentation. For example, while factoring in part of speech can reveal high-frequency part-of-speech patterns like "numbers / letters" and "nouns," thereby increasing the likelihood that "50kv" and "power station" will be identified as new words, the overweighting of part-of-speech features can lead to cases like "1993" and "year" being incorrectly classified as new. This can be caused by factors other than incorrect proportion distribution and a large disparity in granularity. For example, part of speech generally has only a few dozen attributes, such as "noun," "verb," ​​and adjective," while the granularity of word segmentation can reach tens of thousands. Overly broad features can also introduce noise.

[0008] On the other hand, although some technologies combine one or two features, their proportion distribution still relies on manual adjustment. As the number of features increases, the complexity of adjusting parameters will increase exponentially.

[0009] Finally, most current new word discovery evaluation metrics use a Top K evaluation model. Specifically, after a new word set S is mined by a new word discovery algorithm, the top k words with the highest confidence or scores are manually determined to determine the proportion of reasonable words (not limited to metrics such as F1, recall, or precision). This type of evaluation method often includes many highly scored but already common words, such as the word "very," which, despite having a high score, does not contribute much to new word discovery. Summary of the Invention

[0010] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a new word discovery method that combines word vector multi-feature fusion with the characteristics of existing methods being limited to a single or a small number of features, the granularity between features is greatly different, and the feature parameters are difficult to adjust accurately. It combines as many features as possible (part of speech, dependency syntax) and other statistical language features to improve the accuracy of new word mining; clusters the word vectors, adds cluster center features, connects the granularity of different features, and improves the quality of new word discovery; small sample supervised learning feature weight coefficients bring greater field-related new word discovery quality; implements a more reasonable new word evaluation method and more effectively mines potential new words.

[0011] The second object of the present invention is to provide a new word discovery system that combines word vector multi-feature fusion.

[0012] A third object of the present invention is to provide a storage medium.

[0013] A fourth object of the present invention is to provide a computing device.

[0014] In order to achieve the above object, the present invention adopts the following technical solutions:

[0015] The present invention provides a new word discovery method combining word vector multi-feature fusion, comprising the following steps:

[0016] Calculation of statistical natural language feature scores: perform feature annotation on the corpus, traverse all attribute pairs, calculate the mutual information matrix and self-information matrix of the attribute pairs, and calculate the attribute pair scores of all candidate features;

[0017] Calculate the score based on word vector features: Perform k-means clustering on the word vectors. According to the dictionary corresponding to the word vectors, find the corresponding word vectors for all word segments in the corpus, and then find the corresponding nearest neighbor class center vector. Find the word corresponding to the class center vector as the word vector feature attribute, traverse all word vector feature attributes, calculate the self-information matrix and mutual information matrix of the word vector feature attribute pairs, and calculate the scores of all word vector feature attribute pairs;

[0018] After configuring weight coefficients for the attribute pairs of candidate features, sum them up, optimize the weight coefficients through logistic regression, and calculate the total score;

[0019] Initialize the weight coefficient and set the new word threshold. When the total score exceeds the new word threshold, set the random probability to extract the score vector corresponding to the candidate word pair to the sample set, and mark the sample set to obtain the label set;

[0020] Update the weight coefficient based on stochastic gradient descent (SGD), iterate the training until the loss function converges, and obtain the final weight coefficient;

[0021] Recalculate the total scores of all candidate word pairs and output new words that exceed the new word threshold.

[0022] As a preferred technical solution, the corpus is marked with features, and the features include segmentation, part of speech and dependency syntactic relationship.

[0023] As a preferred technical solution, the attribute pair scores of all candidate features are calculated as follows:

[0024] MS m [x i ][x j ]=α m MI m [x i ][x j ]+β m MH m [x i ][x j ]

[0025] Among them, MS m [x i ][x j ] represents the attribute pair score of the candidate feature, MIm [x i ][x j ] represents the mutual information matrix of attribute pairs, MH m [x i ][x j ] represents the self-information matrix of the attribute pair, α m represents the mutual information ratio parameter of the mth feature, β m The self-information ratio parameter of the m-th feature.

[0026] As a preferred technical solution, the scores of all candidate class center pairs are calculated, which is expressed as:

[0027] MS[v i ][v j ]=αMI[v i ][v j ]+βMH[v i ][v j ]

[0028] Among them, MS[v i ][v j ] represents the score of the candidate class center pair, MI[v i ][v j ] represents the mutual information matrix of the class center pair, MH[v i ][v j ] represents the self-information matrix of the class center pair, α represents the mutual information ratio parameter, and β represents the self-information ratio parameter.

[0029] As a preferred technical solution, the step of calculating the word vector feature score is repeated by setting different random seeds during clustering to obtain multiple score matrices.

[0030] As a preferred technical solution, the total score is calculated using the following formula:

[0031]

[0032] in, <x i , x j > represents a candidate word pair, F m (x) indicates the mth feature annotation of word segmentation x, M is the number of statistical natural language features, C is the number of clustering, MS m represents the score of the candidate feature, λ m represents the weight coefficient, and M+C represents the number of features.

[0033] In order to achieve the above second purpose, the present invention adopts the following technical solutions:

[0034] A new word discovery system that combines word vector multi-feature fusion, including: a statistical natural language feature score calculation module, a word vector feature score calculation module, a total score calculation module, a threshold judgment module, an iterative training module, and a new word output module;

[0035] The statistical natural language feature score calculation module is used to calculate the statistical natural language feature score: feature annotate the corpus, traverse all attribute pairs, calculate the mutual information matrix and self-information matrix of the attribute pairs, and calculate the attribute pair scores of all candidate features;

[0036] The word vector feature score calculation module is used to calculate the word vector feature score: perform k-means clustering on the word vector, find the corresponding word vector for all word segments in the corpus according to the dictionary corresponding to the word vector, and then find the corresponding nearest neighbor class center vector, and find the word corresponding to the class center vector as the word vector feature attribute, traverse all word vector feature attributes, calculate the self-information matrix and mutual information matrix of the word vector feature attribute pair, and calculate the score of all word vector feature attribute pairs;

[0037] The total score calculation module is used to configure weight coefficients for the attribute pair scores of the candidate features and then sum them up, optimize the weight coefficients through logistic regression, and calculate the total score;

[0038] The threshold judgment module is used to initialize the weight coefficient and set the new word threshold. When the total score exceeds the new word threshold, the score vector corresponding to the candidate word pair is extracted to the sample set with a random probability, and the sample set is marked to obtain a label set.

[0039] The iterative training module is used to update the weight coefficient based on stochastic gradient descent (SGD), and iterative training is performed until the loss function converges to obtain the final weight coefficient;

[0040] The new word output module is used to recalculate the total scores of all candidate word pairs and output new words that exceed the new word threshold.

[0041] In order to achieve the third purpose above, the present invention adopts the following technical solutions:

[0042] A computer-readable storage medium stores a program, which, when executed by a processor, implements the new word discovery method combined with multi-feature fusion of word vectors as described above.

[0043] In order to achieve the fourth purpose above, the present invention adopts the following technical solutions:

[0044] A computing device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, it implements the new word discovery method combined with multi-feature fusion of word vectors as described above.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] (1) The present invention proposes to start from the breadth of features and try to mine more effective features (word frequency and mutual information, part of speech, dependency syntax, word vector clustering) to improve the accuracy of new word mining.

[0047] (2) The present invention clusters word vectors to obtain features with cluster centers as attributes, further bridging the gap between coarse-grained features and overly fine-grained features, and improving the quality of new word discovery.

[0048] (3) The present invention proposes a small sample supervised learning feature weight coefficient. Compared with the existing technology that generally adopts an unsupervised approach, it uses a small amount of manually labeled samples to bring greater domain-related new word discovery quality.

[0049] (4) The present invention proposes the following evaluation criteria for new words: after subtracting the common word set (from an externally provided common word dictionary) from the candidate word set, the proportion of reasonable words in the first k words is verified (not limited to indicators such as f1, recall or precision), and more attention is paid to the degree of "newness" of the new words. The new word evaluation method is more reasonable and more effective in discovering potential new words. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of the process of the new word discovery method combining word vector multi-feature fusion of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] Example 1

[0053] This embodiment provides a new word discovery method that combines word vector multi-feature fusion, wherein the general specific features include: word segmentation, part of speech and dependency syntactic analysis, etc. For the above commonly used statistical natural language features, most current natural language processing tools can obtain them. In addition, this embodiment will introduce word vector clustering centers as new feature representations. Finally, this embodiment proposes a method for optimizing the weight coefficients between each feature using a machine learning model. Therefore, the specific implementation will be divided into the following two feature evaluation parts: statistical natural language feature evaluation, word vector feature representation evaluation and a weight optimization part: machine learning optimization feature weight coefficient part.

[0054] like Figure 1 As shown, the specific steps include:

[0055] S1: Calculate the statistical natural language feature score. Assume there are M features, including word segmentation, part of speech, and dependency syntactic relationships. For the mth feature, first use a natural language processing tool (such as jiaba or hanlp) to annotate the corpus X with its part of speech and dependency syntactic relationship features. After word segmentation, all candidate word pairs are annotated with features. These annotated pairs are called attribute pairs.

[0056] Then set the self-information matrix Mutual Information Matrix Score Matrix Where K refers to the attribute set S m The size of (for example, the part-of-speech set of hanlp includes adjectives, nouns, etc.).

[0057] Then, iterate over all attribute pairs Calculate its mutual information matrix and self-information matrix. Finally, calculate the performance matrix, that is, the attribute pairs of all candidate features Score (α m / β m is the mutual information / self-information ratio parameter of the mth feature):

[0058] MS m [x i ][x j ]=α m MI m [x i ][x j ]+β m MH m [x i ][x j ]

[0059] S2: Calculation based on word vector feature scores. First, perform k-means clustering on the word vectors with a cluster center vector of K, and denote the cluster center vector as S. K , recommended parameters Where N is the number of vectors in the word vector. Then, according to the dictionary corresponding to the word vector, find the corresponding word vector υ for all the word segments x in the corpus X, and further find the corresponding nearest neighbor class center vector. The word corresponding to the class center vector in the dictionary is the word vector feature attribute of the word segment: Then set the self-information matrix Mutual Information Matrix Scoring Matrix (K refers to the number of clusters). Then, traverse all word vector feature attribute pairs Calculate its self-information matrix and mutual information matrix. Finally, calculate all candidate word vector feature attribute pairs The score (α / β are the parameters of the mutual information / self-information ratio of the word vector):

[0060] MS[v i ][v j ]=αMI[v i ][v j ]+βMH[v i ][v j ]

[0061] Among them, MS[v i ][v j ] represents the score of the candidate word vector feature attribute pair, MI[v i ][v j ] represents the mutual information matrix of word vector feature attribute pairs, MH[v i ][v j ] represents the self-information matrix of the word vector feature attribute pair, α represents the mutual information ratio parameter, and β represents the self-information ratio parameter.

[0062] Step S2 can be repeated multiple times with different random seeds set during clustering to obtain multiple performance matrices to improve a certain effect.

[0063] S3: Optimize weight coefficients through logistic regression. Existing new word discovery methods based on multi-feature fusion all use manual adjustment coefficients. As the number of features increases, the number of weight coefficients also increases proportionally. For parameter adjustment work, the workload increases exponentially, which is unacceptable for limited manual resources. To a certain extent, labeling a small amount of training samples is not only easier than manually adjusting parameters, but also brings higher accuracy. This embodiment proposes a method for adjusting weight coefficients through a deep learning framework. The specific steps are as follows:

[0064] First, when λ is randomly initialized, all candidate word pairs The total score function have:

[0065]

[0066] in, <x i , x j > represents a candidate word pair, F m(x) Indicates that the mth feature annotation is performed on the word segment x, M is the number of statistical natural language features, C is the number of clustering (word vector features are also considered as a feature annotation method), MS m represents the score of the candidate feature, λ m represents the weight coefficient, and M+C represents the number of features.

[0067] For example: F1("artificial") = "artificial", F2("artificial") = "industrious", F3("artificial") = "a". Here, F1 is a word segmentation feature, and the corresponding feature is the word itself, so the word is directly returned. F2 is a mapping of the class center vector, and the word "industrious" corresponding to the class center vector closest to the "artificial" word vector is returned as an attribute label. F3 is a mapping of the word nature, and "artificial" belongs to an adjective, so "a" is returned.

[0068] Set a new word determination threshold δ. If the total score exceeds the new word determination threshold δ, then the score vector s corresponding to its candidate word pair is extracted with a certain random probability and added to the sample set χ data (Generally, only about 100 to 200 training samples need to be extracted), where the score vector s m = MS m [F m (x i )][F m (x j )]:

[0069] In this embodiment, each s in the score vector s is the score of a single feature. For example, s = [score of word segmentation, score of word nature tagging, …, score of word vector clustering], but the weight coefficient of this score is not included here, and this weight coefficient will be added for optimization.

[0070] Furthermore, manually label the new words in χ data , and set the corresponding label set y data for χ label . For the sample x ∈ χ data , if the manual label is correct, then y label [x] = 1, otherwise y label [x] = 0. Set the prediction function f λ (x) as logistic regression.

[0071] Furthermore, set the learning rate μ, generally 0.1, 0.01, etc.;

[0072] Construct a loss function where, y is the label corresponding to the sample x in the label set y label .

[0073] Furthermore, use stochastic gradient descent SGD to update the weight coefficient λ.

[0074] Repeat the above training until the loss function converges, obtain the final weight coefficient λ, recalculate the total score according to the weight coefficient, and then screen out new words with the new word determination threshold;

[0075] In this embodiment, for all candidate word pairs According to the formula s m =MS m [F m (x i )][F m (x j )] Calculate the score vector s and input it as a parameter to f λ (x), if f λ If (x)>δ (new word determination threshold δ), it is added to the new word set S as a new word. Finally, the final new word set S is output.

[0076] In this embodiment, a specific example is used to illustrate that given the White Paper on China's Actions in Fighting the COVID-19 Epidemic, there is the corpus "Currently, the novel coronavirus is still spreading around the world, and the international community will face even more severe difficulties", as well as a natural language processing tool F, a word vector V, and a corresponding dictionary D.

[0077] 1. Use Tool F to annotate the corpus. For example, segmentation (basic features): Currently, the coronavirus is still spreading globally, and the international community will face even more severe difficulties...; part-of-speech features: t / a / n / n / d / p / n / v / v / w / n / n / d / v / v / d / a / u / n..., etc.

[0078] 2. For the above M features, calculate their self-information matrix MH, mutual information matrix MI and score matrix MS respectively;

[0079] 3. Cluster all word vectors V with K=100 to obtain 100 cluster center vectors V 100 .

[0080] 4. For the corpus after word segmentation, such as: current / , / new / corona / virus / is / still / in / the world, it is mapped to the corresponding word vector through the given dictionary D, such as: υ9←"current" (here the 9th word vector in V corresponding to the word "current" is found through the dictionary D).

[0081] 5. Further, by traversing V 100 Find the class center vector corresponding to the word vector after the above mapping, and find the word corresponding to the class center vector as the attribute. For example, the nearest neighbor class center vector of υ9 is V 100 In υ3, the corresponding word found in the dictionary is "now", so the attribute of "current" is "now".

[0082] 6. For the above word vector features, perform clustering 3 times (C=3) based on different random initialization states to obtain different cluster center sets, and repeat steps 4 and 5 after each clustering to calculate the self-information matrix MH, mutual information matrix MI and score matrix MS after each clustering.

[0083] 7. Set the score weight coefficients of feature segmentation, part of speech, cluster 1, cluster 2 and cluster 3 to 0.4, 0.3, 0.1, 0.1 and 0.1 respectively; set the score threshold to 0.1. Suppose that in the end, the scores of candidate pairs such as <new, crown>, <crown, virus>, <international, society>, etc. exceed the threshold.

[0084] Their corresponding score vectors are [0.25, 0.19, 0.11, 0.06, 0.09], [0.13, 0.53, 0.04, 0.11, 0.02], and [0.27, 0.29, 0.09, 0.06, 0.08]. They were manually labeled as [1, 0, 1]. After labeling approximately 100 new words, logistic regression was used to optimize the weight coefficients to [0.55, 0.025, 0.192, 0.013, 0.22].

[0085] 8. Using the optimized weight coefficients, we recalculated the weighted scores for <novel, crown>, <crown, virus>, <international, society>, etc., with scores of 0.18, 0.098, 0.19, etc., and re-output the new word sets of <novel, crown>, <international, society>, etc.;

[0086] As shown in Table 1 below, this embodiment verifies the new word discovery method of the present invention on the text data of the "White Paper on China's Action to Fight the COVID-19 Epidemic" and the "White Paper on Artificial Intelligence Standardization". As shown in Tables 2 to 5 below, the method of the present invention has significant improvements in f1-score, recall rate and precision compared with other methods, and can therefore more effectively mine potential new words (unlike traditional evaluation criteria, the Jieba word segmentation tool is used here to remove common words, so the indicators are numerically lower than previous methods).

[0087] Table 1 Experimental corpus sources and statistical information

[0088] Corpus name field Word count Number of participles Average word frequency White Paper on China's Actions in Fighting the COVID-19 Epidemic politics 37620 3490 5.73 White Paper on Artificial Intelligence Standardization science and technology 53562 3781 7.15

[0089] Table 2 Comparison of new word discovery results in the White Paper on China's Actions in Fighting the COVID-19 Epidemic

[0090] Word segmentation method precision recall f1-score jiagu 15.2 39 21.9 smooth nlp 12.6 35.5 18.6 Method of using the present invention (without optimizing weight coefficient) 16 40 22.9 Method of using the present invention (logistic regression optimization weight coefficient) 22.6 47.5 30.6

[0091] Table 3 Comparison of new word discovery results in the artificial intelligence standardization white paper

[0092]

[0093]

[0094] Table 4: New terms in the White Paper on China's Action to Fight the COVID-19 Epidemic under the use of the present invention

[0095]

[0096] Table 5 New words in the white paper on China's actions to combat the COVID-19 epidemic using the smooth NLP method

[0097]

[0098] Example 2

[0099] This embodiment provides a new word discovery system that combines word vector multi-feature fusion, including: a statistical natural language feature score calculation module, a word vector feature score calculation module, a total score calculation module, a threshold judgment module, an iterative training module, and a new word output module;

[0100] In this embodiment, the statistical natural language feature score calculation module is used to calculate the statistical natural language feature score: feature annotate the corpus, traverse all attribute pairs, calculate the mutual information matrix and self-information matrix of the attribute pairs, and calculate the attribute pair scores of all candidate features;

[0101] In this embodiment, the word vector feature score calculation module is used to calculate the score based on the word vector feature: perform k-means clustering on the word vector, find the corresponding word vector for all word segments in the corpus according to the dictionary corresponding to the word vector, and then find the corresponding nearest neighbor class center vector, and find the word corresponding to the class center vector as the word vector feature attribute, traverse all word vector feature attributes, calculate the self-information matrix and mutual information matrix of the word vector feature attribute pairs, and calculate the scores of all word vector feature attribute pairs;

[0102] In this embodiment, the total score calculation module is used to configure weight coefficients for the scores of the attribute pairs of candidate features (including word vector feature attribute pairs) and then sum them up, optimize the weight coefficients through logistic regression, and calculate the total score;

[0103] In this embodiment, the threshold judgment module is used to initialize the weight coefficient and set the new word threshold. When the total score exceeds the new word threshold, the score vector corresponding to the candidate word pair is extracted to the sample set with a random probability, and the sample set is marked to obtain a label set.

[0104] In this embodiment, the iterative training module is used to update the weight coefficient based on stochastic gradient descent (SGD), and iterative training is performed until the loss function converges to obtain the final weight coefficient;

[0105] In this embodiment, the new word output module is used to recalculate the total scores of all candidate word pairs and output new words that exceed the new word threshold.

[0106] Example 3

[0107] This embodiment provides a storage medium, which may be a ROM, RAM, disk, CD or other storage medium. The storage medium stores one or more programs. When the program is executed by the processor, the new word discovery method combining word vector multi-feature fusion of embodiment 1 is implemented.

[0108] Example 4

[0109] This embodiment provides a computing device, which can be a desktop computer, a laptop computer, a smart phone, a PDA handheld terminal, a tablet computer or other terminal device with a display function. The computing device includes a processor and a memory, and the memory stores one or more programs. When the processor executes the program stored in the memory, the new word discovery method combining word vector multi-feature fusion of Example 1 is implemented.

[0110] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A new word discovery method combining word vector multi-feature fusion, characterized by: The steps include: Calculation of statistical natural language feature scores: perform feature annotation on the corpus, traverse all attribute pairs, calculate the mutual information matrix and self-information matrix of the attribute pairs, and calculate the attribute pair scores of all candidate features, including: For the mth feature, natural language processing tools are used to annotate the corpus X with parts of speech and dependency syntactic relationship features, and all candidate word pairs after word segmentation are annotated with features. The annotated pairs are called attribute pairs. Setting the self-information matrix Mutual Information Matrix Scoring Matrix Where K refers to the attribute set S m size; Iterate over all attribute pairs <x i ,x j >, Calculate its mutual information matrix and self-information matrix, and finally calculate the performance matrix, that is, the attribute pairs of all candidate features <x i ,x j >, The score is specifically expressed as: MS m [x i ][x j ]=α m MI m [x i ][x j ]+β m MH m [x i ][x j ] Among them, MS m [x i ][x j ] represents the attribute pair score of the candidate feature, MI m [x i ][x j ] represents the mutual information matrix of attribute pairs, MH m [x i ][x j ] represents the self-information matrix of the attribute pair, α m represents the mutual information ratio parameter of the mth feature, β m The self-information ratio parameter of the mth feature; Calculation of word vector feature scores: Perform k-means clustering on the word vectors. According to the dictionary corresponding to the word vectors, find the corresponding word vectors for all word segments in the corpus, and then find the corresponding nearest neighbor class center vector. Find the word corresponding to the class center vector as the word vector feature attribute, traverse all word vector feature attributes, calculate the self-information matrix and mutual information matrix of the word vector feature attribute pairs, and calculate the scores of all word vector feature attribute pairs, including: Cluster the word vectors into categories K, and record the cluster center vector as the set S K ; According to the dictionary corresponding to the word vector, find the corresponding word vector v for all the word segments x in the corpus X, find the corresponding nearest neighbor class center vector, and record the word corresponding to the class center vector in the dictionary as the word vector feature attribute of the word segment: Traverse all word vector feature attribute pairs <v i , v j >, Calculate its self-information matrix and mutual information matrix, and calculate all candidate word vector feature attribute pairs <v i ,v j >, The score is expressed as: MS[v i ][v j ]=αMI[v i ][v j ]+βMH[v i ][v j ] Among them, MS[v i ][v j ] represents the score of the candidate word vector feature attribute pair, MI[v i ][v j ] represents the mutual information matrix of word vector feature attribute pairs, MH[v i ][v j ] represents the self-information matrix of the word vector feature attribute pair, α represents the mutual information ratio parameter, and β represents the self-information ratio parameter; After configuring the weight coefficients for the scores of all candidate feature attribute pairs, the sum is calculated, and the weight coefficients are optimized through logistic regression to calculate the total score. When λ is randomly initialized, all candidate word pairs <x i ,x j >, The total score function have: Among them, F m (x) indicates the mth feature annotation of word segmentation x, M is the number of statistical natural language features, C is the number of clustering, MS m represents the score of the candidate feature, λ m represents the weight coefficient, M+C represents the number of features; Initialize the weight coefficient and set the new word threshold. When the total score exceeds the new word threshold, set the random probability to extract the score vector corresponding to the candidate word pair to the sample set, and mark the sample set to obtain the label set; Update the weight coefficient based on stochastic gradient descent (SGD), iterate the training until the loss function converges, and obtain the final weight coefficient; Recalculate the total scores of all candidate word pairs and output new words that exceed the new word threshold.

2. The new word discovery method combining word vector multi-feature fusion according to claim 1 is characterized in that: The corpus is marked with features, wherein the features include segmentation, part of speech and dependency syntactic relationship.

3. The new word discovery method combining word vector multi-feature fusion according to claim 1 is characterized in that: The step of calculating the word vector feature score is repeated by setting different random seeds during clustering to obtain multiple score matrices.

4. A new word discovery system combining word vector multi-feature fusion, characterized by: A method for discovering new words by combining word vector multi-feature fusion according to any one of claims 1 to 3, comprising: a statistical natural language feature score calculation module, a word vector feature score calculation module, a total score calculation module, a threshold judgment module, an iterative training module, and a new word output module; The statistical natural language feature score calculation module is used to calculate the statistical natural language feature score: feature annotate the corpus, traverse all attribute pairs, calculate the mutual information matrix and self-information matrix of the attribute pairs, and calculate the attribute pair scores of all candidate features; The word vector feature score calculation module is used to calculate the word vector feature score: perform k-means clustering on the word vector, find the corresponding word vector for all word segments in the corpus according to the dictionary corresponding to the word vector, and then find the corresponding nearest neighbor class center vector, and find the word corresponding to the class center vector as the word vector feature attribute, traverse all word vector feature attributes, calculate the self-information matrix and mutual information matrix of the word vector feature attribute pair, and calculate the score of all word vector feature attribute pairs; The total score calculation module is used to configure weight coefficients for the attribute pair scores of the candidate features and then sum them up, optimize the weight coefficients through logistic regression, and calculate the total score; The threshold judgment module is used to initialize the weight coefficient and set the new word threshold. When the total score exceeds the new word threshold, the score vector corresponding to the candidate word pair is extracted to the sample set with a random probability, and the sample set is marked to obtain a label set. The iterative training module is used to update the weight coefficient based on stochastic gradient descent (SGD), and iterative training is performed until the loss function converges to obtain the final weight coefficient; The new word output module is used to recalculate the total scores of all candidate word pairs and output new words that exceed the new word threshold.

5. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, a new word discovery method combining word vector multi-feature fusion as described in any one of claims 1 to 3 is implemented.

6. A computing device comprising a processor and a memory for storing a program executable by the processor, characterized in that When the processor executes the program stored in the memory, it implements the new word discovery method combined with word vector multi-feature fusion as described in any one of claims 1-3.

Images