A target new word determination method and device based on a cohesion degree algorithm and a medium

Abstract

The present application relates to the technical field of data processing, and particularly relates to a target new word determination method and device based on a cohesion degree algorithm and a medium, the method comprising: processing a given corpus to obtain a plurality of candidate new words, calculating a plurality of attribute indexes of each candidate new word, dynamically obtaining an index evaluation threshold of each attribute index of the candidate new word according to the length of the candidate new word, a frequency adjustment factor corresponding to the candidate new word, and the numerical distribution of each attribute index corresponding to the plurality of candidate new words in the given corpus, marking 0 when any attribute index is less than the corresponding index evaluation threshold, and marking 1 otherwise, and determining the candidate new word as a target new word when the product of all the marked values corresponding to the candidate new word is 1; the present application improves the accuracy and reliability of target new word screening by obtaining a plurality of dimension attribute indexes of the candidate new word and dynamically adjusting the index evaluation threshold of each attribute index of each candidate new word.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a target new word determination method and device based on a cohesion degree algorithm and a medium. BACKGROUND

[0002] In the field of natural language processing, keyword extraction is a basic link for text mining and information retrieval tasks. Traditional keyword extraction methods based on dictionaries rely heavily on pre-constructed dictionary libraries. However, with the development of society, new words such as network language, professional terms, and emerging entities are emerging in an endless stream. These new words cannot be effectively identified because they are not included in existing dictionaries, resulting in missing keywords, so the knowledge base of the keywords needs to be supplemented.

[0003] At present, in order to solve the above problems, an unsupervised learning algorithm based on statistical features is usually used to automatically identify new words from large-scale corpus. Among them, the SmoothNLP new word discovery algorithm is a common solution. The cohesion degree algorithm is used as the core cornerstone. Generally, the internal cohesion of the candidate string such as point mutual information and boundary freedom is calculated, and a globally fixed discrimination threshold is set to determine whether a string constitutes an effective new word. However, the accuracy of new word screening is affected by the failure to fully exploit the features of new words and corpus and the use of a fixed discrimination threshold. SUMMARY

[0004] In view of the above technical problems, the present application provides a target new word determination method and device based on a cohesion degree algorithm and a medium. By obtaining multiple dimension attribute indicators of candidate new words and dynamically adjusting the indicator evaluation threshold of each attribute indicator of each candidate new word, the accuracy and reliability of target new word screening are improved.

[0005] According to a first aspect of the present application, a target new word determination method based on a cohesion degree algorithm is provided, comprising the following steps:

[0006] S1, processing a given corpus to obtain a plurality of candidate new words.

[0007] S2, for any candidate new word, calculating a plurality of attribute indicators of the candidate new word; the plurality of attribute indicators include average mutual information, mutual information, sum of left information entropy and right information entropy based on the cohesion degree algorithm, and heat score, wherein the heat score is obtained by weighting calculation according to the average mutual information, the sum of left information entropy and right information entropy, and the corresponding target weight respectively.

[0008] S3, dynamically obtaining the indicator evaluation threshold of each attribute indicator of the candidate new word according to the length of the candidate new word, the frequency adjustment factor corresponding to the candidate new word obtained in advance, and the numerical distribution of each attribute indicator corresponding to a plurality of candidate new words in the given corpus.

[0009] S4, when any attribute index is less than the corresponding index evaluation threshold, mark 0, otherwise mark 1, and when the product of all mark values corresponding to the candidate new word is 1, determine the candidate new word as the target new word.

[0010] According to a second aspect of the present application, a non-transitory computer readable storage medium is provided, the storage medium storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement the above-mentioned target new word determination method based on the cohesion degree algorithm.

[0011] According to a third aspect of the present application, an electronic device is provided, comprising a processor and the above-mentioned non-transitory computer readable storage medium.

[0012] The present application has at least the following beneficial effects:

[0013] The present application provides a target new word determination method based on a cohesion degree algorithm, first processing a given corpus to obtain a plurality of candidate new words, and calculating a plurality of attribute indexes of each candidate new word, which can grasp the feature information of the candidate new word in multiple dimensions, and is beneficial to analyze the candidate new word from multiple aspects; then according to the length of the candidate new word, the frequency adjustment factor corresponding to the candidate new word and the numerical distribution of each attribute index corresponding to a plurality of candidate new words in the given corpus, the index evaluation threshold of each attribute index of the candidate new word is dynamically obtained, when any attribute index is less than the corresponding index evaluation threshold, mark 0, otherwise mark 1, and when the product of all mark values corresponding to the candidate new word is 1, determine the candidate new word as the target new word, by dynamically adjusting the index evaluation threshold of each attribute index of each candidate word according to the length of the candidate new word and the statistical information of the given corpus, the accuracy and reliability of the target new word screening is improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0015] Figure 1 The flow chart of the target new word determination method based on the cohesion degree algorithm provided by the embodiments of the present application. DETAILED DESCRIPTION

[0016] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0017] The present application provides a target new word determination method based on a cohesion degree algorithm, as shown in the formula (1), the method comprises the following steps: Figure 1

[0018] S1, processing a given corpus to obtain a plurality of candidate new words.

[0019] Specifically, the S1 step comprises the following steps:

[0020] S101, extracting all n-grams from the given corpus as first words to be processed; wherein the value range of n is 2-k, and k is a preset maximum number of characters. For example, k=7, and all 2-7 character groups are extracted. In a specific implementation, an N-gram model can be used for extraction.

[0021] S102, performing part-of-speech tagging on each first word to be processed, and filtering out a plurality of first words to be processed that do not meet the preset part-of-speech requirements from the plurality of first words to be processed, to obtain a plurality of second words to be processed. For example, according to the key word expansion business requirement, pronouns, adjectives, etc. are filtered out.

[0022] S103, based on a preset common word library, traversing the plurality of second words to be processed and filtering out the second words to be processed that belong to common words, to finally obtain a plurality of candidate new words.

[0023] Firstly, all continuous character combinations of n-grams are extracted, and a first filtering based on part-of-speech is performed on the business requirement, to preliminarily reduce the number of candidate new words. Then, a second filtering is performed by filtering out common words, so that the obtained candidate new words are closer to new words, and the accuracy of the candidate new words is improved.

[0024] S2, for any candidate new word, calculating a plurality of attribute indexes of the candidate new word; the plurality of attribute indexes comprise average mutual information, mutual information, sum of left information entropy and right information entropy based on the cohesion degree algorithm, and a hotness score, wherein the hotness score is obtained by weighting calculation according to the average mutual information, the sum of the left information entropy and the right information entropy, and the corresponding target weights respectively. It should be noted that the average mutual information refers to the average value of a plurality of point mutual informations corresponding to the n-grams, and the point mutual information is calculated by the cohesion degree algorithm. Those skilled in the art know the specific calculation method of the average mutual information, the mutual information, and the sum of the left information entropy and the right information entropy, which will not be described here.

[0025] ​As mentioned above, by acquiring several attribute indicators of candidate new words, we can grasp the feature information of candidate new words in multiple dimensions. Moreover, each attribute indicator introduced is related to the evaluation of new words, which is conducive to analyzing candidate new words from multiple aspects and making the final selected target new words more accurate and reliable.

[0026] In one implementation, the target weights corresponding to the average mutual information, the sum of the left information entropy and the right information entropy are obtained through the following steps:

[0027] S201: For several candidate new word samples obtained in advance from the preset corpus validation set, the average mutual information, the sum of the left information entropy and the right information entropy of each candidate new word sample are normalized to between 0 and 1. For example, Min-Max normalization or Z-score standardization can be used for normalization processing, which will not be elaborated here.

[0028] S202, based on the preset weights corresponding to the average mutual information and the weights corresponding to the sum of the left and right information entropies, several weight combinations are obtained. For example, if the average mutual information corresponds to 'a' preset weights and the sum of the left and right information entropies corresponds to 'b' preset weights, then there are a × b weight combinations in total.

[0029] S203: Based on the sum of the average mutual information, left information entropy, and right information entropy of the normalized candidate new word samples, calculate the corresponding popularity score for each weight combination. Combine the average mutual information, mutual information, left information entropy, and right information entropy of the candidate new word samples to select several target new word samples for each weight combination from the candidate new word samples. This can be understood as: selecting target new word samples according to several attribute indicators of the candidate new word samples, following the implementation method of steps S3-S4.

[0030] S204: Based on the pre-labeled actual target new words and several target new word samples under each weight combination, calculate the corresponding F1 score under each weight combination, and obtain the target weights corresponding to the average mutual information, the sum of left information entropy and right information entropy based on the weight combination corresponding to the highest F1 score.

[0031] Specifically, the F1 score for any weight combination meets the following conditions:

[0032] F1 = 2P × R / (P + R), where P is the accuracy obtained from a number of target new word samples and pre-labeled actual target new words under any weight combination, and R is the recall obtained from a number of target new word samples and pre-labeled actual target new words under any weight combination. Those skilled in the art are familiar with the calculation methods of P and R, and will not elaborate further here.

[0033] As mentioned above, when obtaining target weights, the principle of grid optimization algorithm is introduced to traverse the popularity scores obtained under different weight combinations. Based on the actual target new words and the target new word samples under any weight combination, the corresponding F1 score under any weight combination is calculated, which balances precision and recall and makes the obtained weight combination more reliable.

[0034] In another implementation, the target weights corresponding to the average mutual information, the sum of the left information entropy and the right information entropy are obtained through the following steps:

[0035] S210: Based on the average word frequency and word repetition rate of the given corpus, obtain the word sparsity of the given corpus.

[0036] Specifically, the average word frequency A meets the following condition: A = Z / Z0, where Z is the total number of words in the given corpus and Z0 is the number of unique words in the given corpus.

[0037] Specifically, the word repetition rate B meets the following condition: B=(Z-Z0) / Z.

[0038] Furthermore, the word sparsity S of the given corpus meets the following condition:

[0039] S = w1 × (1 - A0) + w2 × (1 - B), where A0 is the normalized value of the average word frequency, B is the word repetition rate, and w1 and w2 are the preset weights of A0 and B, respectively.

[0040] Furthermore, A0 = min(1, A / G), where G is the preset maximum word frequency.

[0041] S220, based on the mapping relationship between the word sparsity of the given corpus and the weight of the average mutual information, determine the target weight corresponding to the average mutual information; wherein, the word sparsity is positively correlated with the weight of the average mutual information. In specific implementations, the mapping relationship can be a linear relationship, which can be set by those skilled in the art according to actual needs.

[0042] S230, Based on the target weight corresponding to the average mutual information, determine the target weight corresponding to the sum of the left information entropy and the right information entropy; wherein, the target weight corresponding to the average mutual information and the target weight corresponding to the sum of the left information entropy and the right information entropy are added together to 1.

[0043] As mentioned above, when the words in the given corpus are relatively sparse, the words are scattered and the information is limited. In this case, we should focus on the internal structure of the words, increase the weight of the average mutual information, and decrease the weight of the sum of the left and right information entropies. When the words are relatively dense, we should increase the weight of the sum of the left and right information entropies and decrease the weight of the average mutual information in order to obtain a more reasonable target weight.

[0044] S3, dynamically obtaining the index evaluation threshold of each attribute index of the candidate new word according to the length of the candidate new word, the frequency adjustment factor corresponding to the candidate new word pre-acquired, and the numerical value distribution of each attribute index corresponding to the candidate new word in the given corpus.

[0045] Preferably, the frequency adjustment factor corresponding to the candidate new word satisfies the following condition:

[0046] λ = min(λ0, max(1, D / f)), wherein λ is the frequency adjustment factor corresponding to the candidate new word, λ0 is a preset maximum frequency adjustment factor, D is a preset word frequency relaxation threshold, and f is the word frequency of the candidate new word. For example, when D is set to 100, the value of max(1, D / f) will become larger when f < 100. Since the appearance frequency of a new word is usually low, the frequency adjustment factor of a low-frequency word is relaxed to prevent missing a real new word. Moreover, λ0 is set to prevent excessive relaxation, so λ0 should be greater than 1.

[0047] Specifically, the S3 step includes the following steps:

[0048] S301, for any candidate new word, determining a length coefficient of the candidate new word according to the length of the candidate new word, and adjusting the preset evaluation threshold of each attribute index of the candidate new word based on the length coefficient to obtain the first evaluation threshold corresponding to each attribute index of the candidate new word; wherein the length coefficient of the candidate new word is negatively correlated with the length of the candidate new word. For example, when the lengths of the candidate new words are 2, 3, 4, and 5, the length coefficients are set to 1.2, 1, 0.8, and 0.6, respectively. Those skilled in the art can make specific settings according to actual needs.

[0049] S302, obtaining a preset numerical quantile corresponding to each attribute index based on the numerical value distribution of each attribute index corresponding to the candidate new words in the given corpus. In one specific embodiment, the preset numerical quantile is the 75th quantile.

[0050] S303, for any candidate new word, adjusting the preset numerical quantile corresponding to the attribute index of the candidate new word based on the frequency adjustment factor corresponding to the candidate new word and a preset attenuation coefficient, and comparing the preset evaluation threshold of each attribute index to obtain the second evaluation threshold corresponding to each attribute index of the candidate new word.

[0051] Further, the S303 step includes the following steps:

[0052] S3031, determining the product of the preset numerical quantile corresponding to each attribute index of the candidate new word and the preset attenuation coefficient as the candidate evaluation threshold of the attribute index itself. The preset attenuation coefficient can be set to 0.7-0.8.

[0053] S3032, the candidate evaluation threshold of mutual information is updated by multiplying the candidate evaluation threshold of mutual information by the frequency adjustment factor corresponding to the candidate new word, to obtain a new candidate evaluation threshold of mutual information.

[0054] S3033, for any attribute index, the maximum of the candidate evaluation threshold of the attribute index and the preset evaluation threshold of the attribute index is determined as the second evaluation threshold corresponding to the attribute index.

[0055] In the above, when obtaining the second evaluation threshold, the preset numerical quantile of the attribute index is obtained first instead of the average value, avoiding the influence of extreme values, and then the preset numerical quantile is attenuated to a certain extent by the preset attenuation coefficient, preventing the obtained threshold from being too high to filter out new words, and also considering the strong correlation between mutual information and word frequency, the candidate evaluation threshold of mutual information is adjusted again, through the above multi-layer processing, the obtained second evaluation threshold is more reasonable and reliable.

[0056] S304, for any attribute index of the candidate new word, the index evaluation threshold of the attribute index of the candidate new word is obtained by calculating the weighted sum according to the first evaluation threshold, the second evaluation threshold, the preset first evaluation threshold weight and the second evaluation threshold weight corresponding to the attribute index. The first evaluation threshold weight and the second evaluation threshold weight are set by the person skilled in the art according to the actual needs, generally considering that the influence of word length on new word screening is less than the influence of corpus statistical information on new word screening, the first evaluation threshold weight and the second evaluation threshold weight can be set to 0.4 and 0.6 respectively.

[0057] In the above, when obtaining the index evaluation threshold, since the lengths of the candidate new words are different, the statistical information of different candidate new words in the given corpus will affect the calculation result of the attribute index value, and the use of a unified index evaluation threshold will cause the reduction of the accuracy of new word screening, therefore, the above method is used to set different index evaluation thresholds for different candidate new words, which is beneficial to improve the screening accuracy of the target new word.

[0058] S4, when any attribute index is less than the corresponding index evaluation threshold, it is marked as 0, otherwise it is marked as 1, and when the product of all the marked values corresponding to the candidate new word is 1, the candidate new word is determined as the target new word; it can be understood that: when each attribute index is not less than the corresponding index evaluation threshold, the candidate new word is regarded as the target new word.

[0059] In the above, when judging whether the candidate new word is the target new word, each attribute index of the candidate new word must be not less than the corresponding index evaluation threshold, and the candidate new word is considered to be the target new word only when each attribute index meets the requirement, which improves the accuracy and reliability of the target new word screening.

[0060] Further, after the step S4, the following steps are further included:

[0061] S10, arranging the target new words in descending order of word length based on the screened target new words, to obtain a sorting result.

[0062] S20, searching whether each target new word is contained in any target new word before the target new word itself in the sorting result, if yes, deleting the target new word itself, otherwise, not deleting the target new word itself. For example, the target new word "wucibaolan" is contained in the target new word "wucibaolanhei", so the target new word "wucibaolan" is deleted, while the target new word "wucibaolanhei" is selected as the final new word.

[0063] S30, determining the remaining target new words in the target new words as the final new words screened based on the deleted target new words.

[0064] As the continuous multi-character is less common than the continuous short character contained therein, when each attribute index of the continuous multi-character is not less than the corresponding index evaluation threshold, the continuous multi-character is more inclined to be a new word than the continuous short character. Therefore, the target new words are filtered again to make the new words more complete.

[0065] The embodiment of the present application further provides a non-transitory computer readable storage medium, which can be arranged in an electronic device to save at least one instruction or at least one program related to a method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to realize the target word screening method provided by the above-mentioned embodiment.

[0066] The embodiment of the present application further provides an electronic device, which comprises a processor and the aforementioned non-transitory computer readable storage medium.

[0067] Although some specific embodiments of the present application have been described in detail by examples, those skilled in the art should understand that the above examples are only for illustration, but not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A method for determining a target new word based on a cohesion algorithm, characterized in that, The method comprises the following steps: S1, processing a given corpus to obtain a plurality of candidate new words; S2, for any candidate new word, calculating a plurality of attribute indicators of the candidate new word; the plurality of attribute indicators include average mutual information based on cohesion degree algorithm, mutual information, sum of left information entropy and right information entropy, and heat score, the heat score is obtained by weighting calculation according to average mutual information, sum of left information entropy and right information entropy, and corresponding target weight respectively; S3, according to the length of the candidate new word, the frequency adjustment factor corresponding to the candidate new word obtained in advance, and the numerical distribution of each attribute indicator corresponding to a plurality of candidate new words in the given corpus, the index evaluation threshold of each attribute indicator of the candidate new word is dynamically obtained; Wherein, the step S3 comprises the following steps: S301, for any candidate new word, the length coefficient of the candidate new word is determined according to the length of the candidate new word, and the preset evaluation threshold of each attribute indicator of the candidate new word is adjusted based on the length coefficient to obtain the first evaluation threshold corresponding to each attribute indicator of the candidate new word; wherein, the length coefficient of the candidate new word is negatively correlated with the length of the candidate new word; S302, based on the numerical distribution of each attribute indicator corresponding to a plurality of candidate new words in the given corpus, the preset numerical quantile number corresponding to each attribute indicator is obtained; S303, for any candidate new word, the preset numerical quantile number corresponding to the attribute indicator of the candidate new word is adjusted based on the frequency adjustment factor corresponding to the candidate new word and the preset attenuation coefficient, and the preset evaluation threshold of each attribute indicator is compared to obtain the second evaluation threshold corresponding to each attribute indicator of the candidate new word; S304, for any attribute indicator of the candidate new word, the index evaluation threshold of the attribute indicator of the candidate new word is obtained by weighting and calculation according to the first evaluation threshold, the second evaluation threshold, the preset first evaluation threshold weight and the second evaluation threshold weight corresponding to the attribute indicator; S4, when any attribute indicator is less than the corresponding index evaluation threshold, it is marked as 0, otherwise it is marked as 1, and when the product of all the marked values corresponding to the candidate new word is 1, the candidate new word is determined as the target new word. 2.The target new word determination method based on the cohesion degree algorithm according to claim 1, characterized in that, The step S1 comprises the following steps: S101, extracting all n-grams from the given corpus as first to-be-processed words; wherein, the value range of n is 2-k, and k is the preset maximum number of characters; S102, performing part-of-speech tagging on each first to-be-processed word, and filtering out a plurality of first to-be-processed words that do not meet the preset part-of-speech requirement from a plurality of first to-be-processed words to obtain a plurality of second to-be-processed words; S103, based on the preset common word library, traversing a plurality of second to-be-processed words and filtering out second to-be-processed words belonging to common words, finally obtaining a plurality of candidate new words. 3.The target new word determination method based on the cohesion algorithm according to claim 1, characterized in that, In the step S2, the target weights corresponding to the average mutual information, the sum of the left information entropy and the right information entropy are obtained by the following steps: S201, for a plurality of candidate new word samples obtained from the preset corpus verification set, the average mutual information, the sum of the left information entropy and the right information entropy of each candidate new word sample are normalized to 0-1; S202, obtain a plurality of weight combinations according to a plurality of weights corresponding to the average mutual information and a plurality of weights corresponding to the sum of the left information entropy and the right information entropy; S203, calculate a corresponding heat score under each weight combination based on the average mutual information, the sum of the left information entropy and the right information entropy of the candidate new word sample after normalization processing, and screen a plurality of target new word samples under each weight combination from a plurality of candidate new word samples in combination with the average mutual information, the mutual information, and the sum of the left information entropy and the right information entropy of the candidate new word sample; S204, calculate a corresponding F1 score under each weight combination based on the actual target new word pre-labeled and the plurality of target new word samples under each weight combination screened, and obtain target weights corresponding to the average mutual information, the sum of the left information entropy and the right information entropy respectively based on the weight combination corresponding to the highest F1 score. 4.The target new word determination method based on the cohesion algorithm according to claim 1, wherein, In the S2 step, the target weights corresponding to the average mutual information and the sum of the left information entropy and the right information entropy are also obtained by the following steps: S210, obtain the word sparsity degree of the given corpus according to the average word frequency and the word repetition rate corresponding to the given corpus; S220, determine the target weight corresponding to the average mutual information according to the mapping relationship between the word sparsity degree and the weight of the average mutual information of the given corpus; wherein the weight of the average mutual information is positively correlated with the word sparsity degree; S230, determine the target weight corresponding to the sum of the left information entropy and the right information entropy according to the target weight corresponding to the average mutual information; wherein the sum of the target weight corresponding to the average mutual information and the target weight corresponding to the sum of the left information entropy and the right information entropy is 1. 5.The target new word determination method based on the cohesion algorithm according to claim 1, wherein, The frequency adjustment factor corresponding to the candidate new word meets the following conditions: λ = min(λ0, max(1, D / f)), wherein λ is the frequency adjustment factor corresponding to the candidate new word, λ0 is a preset maximum frequency adjustment factor, D is a preset word frequency relaxation threshold, and f is the word frequency of the candidate new word. 6.The target new word determination method based on the cohesion algorithm according to claim 1, wherein, The S303 step includes the following steps: S3031, determine the product of the preset numerical quantile corresponding to each attribute index of the candidate new word and the preset attenuation coefficient as the candidate evaluation threshold of the attribute index itself; S3032, update the candidate evaluation threshold of the mutual information by multiplying the candidate evaluation threshold of the mutual information by the frequency adjustment factor corresponding to the candidate new word and the frequency adjustment factor corresponding to the candidate new word, to obtain a new candidate evaluation threshold of the mutual information; S3033, for any attribute index, determine the maximum of the candidate evaluation threshold of the attribute index and the preset evaluation threshold of the attribute index as the second evaluation threshold corresponding to the attribute index. 7.The target new word determination method based on the cohesion algorithm according to claim 1, characterized in that, After the S4 step, the following steps are further included: S10, arrange the target new words in descending order of word length based on the plurality of target new words screened to obtain a sorting result; S20, traverse the sorting result to find out whether each target new word is contained in any target new word located before the target new word itself in the sorting result, if yes, delete the target new word itself, otherwise, do not delete the target new word itself; S30, determine the remaining target new words in the plurality of target new words as the final new words screened based on the deleted target new words. 8.A non-transitory computer readable storage medium having stored therein at least one instruction or at least one piece of program, characterized in that, The at least one instruction or the at least one program is loaded by the processor and executed to implement the target new word determination method based on the cohesion degree algorithm as claimed in any one of claims 1-7.

9. An electronic device, comprising: The non-transitory computer readable storage medium as claimed in claim 8 is included in the processor.

Images