A hot word extraction method and device

Abstract

The present invention belongs to the field of soft science and technology, and specifically relates to a method and device for extracting hot words. The method first obtains all candidate words, and counts the frequency data of each candidate word in different time periods; for each candidate word, the frequency data is sorted in chronological order to obtain the original sequence, and the frequency data is sorted according to the frequency size relationship to obtain the sorted sequence; then the growth capacity and rearrangement loss of each candidate word are calculated to calculate the popularity of each candidate word, and the candidate words with high popularity are selected as hot words. The method has high accuracy in hot word extraction; moreover, the method is based on the analysis of the sorted sequence, which eliminates the interference of noise fluctuations and further improves the accuracy of hot word extraction; in addition, the method does not rely on any data set, is insensitive to parameters, and has high universality.

Description

Technical Field

[0001] The present invention belongs to the field of soft science technology, and specifically relates to a hot word extraction method and device. Background Art

[0002] Hot words are words whose frequency of use suddenly increases over a period of time. They include words that have existed in the past but have recently increased in frequency, as well as new words that did not exist before. Extracting hot words from text information such as news, Weibo, comments, and scientific papers is an important means to grasp the development dynamics of things, analyze trends, and monitor public opinion.

[0003] At present, the hot word extraction method is mainly based on frequency fluctuation analysis and calculation. Most methods measure the frequency change by the frequency growth rate of the vocabulary in the recent time window and the long-term time window, calculate the rate of frequency change to get the popularity of the word, and select the words with the highest popularity as the hot words. Through research, it was found that there are three main problems facing hot word extraction: First, the definition of hot words often deviates due to different application scenarios, and each algorithm has different degrees of bias towards new words and high-frequency words; second, each algorithm relies heavily on modeling data when selecting model parameters, and the universality and usability of the model are affected to a certain extent; third, the frequency of hot words will have certain noise fluctuations during the rising, falling or oscillating periods of usage. How to eliminate the interference of noise is an important challenge facing hot word extraction.

[0004] The Chinese invention patent with the authorization announcement number CN107967299B discloses a hot word extraction method, which uses three time windows of different lengths: long-term changes, short-term changes, and overall fluctuations. The "Research on Short Text Hot Topic Detection Based on Word Frequency Mean Fluctuation and Probabilistic Language Model" published by Xu Min and Li Guangjian in the 6th issue of Volume 38 of "Intelligence Magazine" in June 2019 adopts the method of taking the average to avoid the drastic fluctuation of a single data point. The "Research on Weibo Hot Word Extraction and Topic Discovery" by Hao Xiaoling, Mao Jiahui, and Yu Xiuyan, published in the 6th issue of Volume 34 of "Intelligence Magazine" in June 2015, adopts the method of time window to cover up the noise by adjusting the length of the time window. The starting point of the above processing methods is to cover up the noise by comprehensively measuring multiple data points, because the noise is considered to appear at very individual points with a high probability. However, the appearance of noise will still greatly affect the hot word extraction results, especially when the noise occurs at the boundary of the window, it is difficult to effectively process, which ultimately results in low hot word extraction accuracy. Summary of the invention

[0005] The present invention provides a hot word extraction method and device, which are used to solve the problems that the hot word extraction method in the prior art is overly dependent on parameter selection, resulting in poor usability, susceptibility to noise, and low extraction accuracy.

[0006] In order to solve the above technical problems, the technical solutions included in the present invention and the beneficial effects corresponding to the technical solutions are as follows:

[0007] A hot word extraction method of the present invention comprises the following steps:

[0008] 1) Obtain all candidate words and count the frequency data of each candidate word in different time periods;

[0009] 2) For the i-th candidate word, sort the frequency data in chronological order to obtain the original sequence V i , sort the frequency data according to the frequency relationship to obtain the sorting sequence V′ i ;

[0010] 3) The following method is used to calculate the growth energy E of the i-th candidate word: i :

[0011] Extract the sorted sequence V′ of the i-th candidate word i different observation groups, one observation group includes two different subsequences, namely a first subsequence and a second subsequence, the maximum value of the frequency data in the first subsequence is greater than or equal to the maximum value of the frequency data in the second subsequence, and the minimum value of the frequency data in the first subsequence is greater than or equal to the minimum value of the frequency data in the second subsequence;

[0012] Calculate the difference between the first subsequence and the second subsequence in an observation group;

[0013] According to the difference between the two subsequences in each observation group, the growth energy E of the i-th candidate word is calculated. i ;

[0014] 4) Calculate the rearrangement loss R of the i-th candidate word using the following method: i :

[0015] The i-th candidate word is taken from the original sequence V i Transform to sorted sequence V′ i During the process, the position interval and frequency difference between two frequency data to be exchanged are determined each time the frequency data is exchanged;

[0016] According to the position interval and frequency difference corresponding to each frequency data exchange, the reordering loss R of the i-th candidate word is calculated i :

[0017] 5) The growth amount of the i-th candidate word can be E i and the rearrangement loss R i Substitute it into the constructed heat model and calculate the heat H of the i-th candidate word i The relationship reflected by the heat model is: the growth energy E iWith heat H i Positively correlated, rearrangement loss R i With heat H i Negative correlation;

[0018] 6) Calculate the heat of each candidate word according to steps 2) to 5), and select candidate words whose heat is greater than the heat threshold as extracted hot words.

[0019] The beneficial effects of the above technical solution are as follows: the present invention first obtains the original sequence and sorted sequence of each candidate word based on the frequency of occurrence of the candidate words, and then analyzes the original sequence and sorted sequence of each candidate word from the two aspects of growth capacity and rearrangement loss, splitting the original analysis of the growth trend of a frequency sequence into two levels, wherein the growth capacity represents the growth amplitude of the frequency data in the sorted sequence, and the rearrangement loss represents the difference between the original sequence and the sorted sequence, and then according to the growth capacity and rearrangement loss, as well as the constructed heat model, the heat of each candidate word can be calculated to select the real hot words, and the hot word extraction accuracy is high. The beneficial effect of this splitting is mainly due to the fact that, without changing the original frequency value, the frequency data in the sorted sequence is in the state of optimal smoothness, which is explained from a mathematical point of view as the first-order difference has the same sign and the second-order difference has the smallest variance. For a relatively smooth and monotonic function curve, the selection of some parameters (such as the length of the observation window, the weights of different time periods) is less affected by noise. Different from Fourier transform denoising, wavelet transform denoising, local mean and other smoothing methods, the method of converting the original sequence into a sorted sequence and retaining the sorting operation does not have any information loss, while the other smoothing methods have different degrees of information loss. Therefore, this method is based on the sorting sequence and sorting operation for analysis, while retaining the original information, it excludes the interference of noise fluctuations, effectively avoids noise, and further improves the accuracy of hot word extraction. In addition, this method does not rely on any data set, is not sensitive to parameters, and has high universality.

[0020] Furthermore, in order to accurately characterize the changes in the two subsequences, in step 3), the difference between the first subsequence and the second subsequence in the one observation group is the growth rate of the frequency corresponding to the first subsequence compared to the frequency corresponding to the second subsequence, and the frequency corresponding to the first subsequence is the average value of all frequency data in the first subsequence, and the frequency corresponding to the second subsequence is the average value of all frequency data in the second subsequence.

[0021] Furthermore, in step 3), the growth energy E of the i-th candidate word is calculated based on the difference between the two subsequences in each observation group. i The method used is: weighted sum the differences between the two subsequences of each observation group to obtain the growth energy E of the i-th candidate word i.

[0022] Furthermore, in step 4), the rearrangement loss R of the i-th candidate word is calculated according to the position interval and frequency difference corresponding to each frequency data exchange. i The method is as follows: multiply the position interval and frequency difference corresponding to a frequency data exchange to obtain the product result corresponding to a frequency data exchange; add the product results corresponding to each frequency data exchange to obtain the reordering loss R of the i-th candidate word i .

[0023] Furthermore, in step 4), in order to remove the i-th candidate word from the original sequence V i Transform to sorted sequence V′ i , the frequency data exchange method used is:

[0024] From the sorted sequence V′ i The frequency data at the first position in the original sequence V is interchanged in the following manner. i With sorted sequence V′ i Same: For sorted sequence V′ i The kth frequency data v′ in i,k , find the first intermediate sequence that satisfies v starting from the kth position after the last frequency data exchange i,j =v′ i,k Frequency data v i,j , and replace the frequency data v in the intermediate sequence obtained after the last frequency data exchange i,j With frequency data v i,k The positions of the two frequencies are swapped to obtain the intermediate sequence after the frequency data is swapped.

[0025] Further, in step 5), the heat model is:

[0026]

[0027] Where m is the original sequence V i Length.

[0028] Furthermore, the method for obtaining candidate words in step 1) is: obtaining various documents and extracting corpus from the documents; performing word segmentation processing on the corpus to obtain word segmentation results; and determining candidate words based on the word segmentation results.

[0029] Furthermore, in order to obtain accurate word segmentation results, the method for determining candidate words according to the word segmentation results includes: performing data cleaning on the word segmentation results to obtain candidate words; the data cleaning method includes at least one of the following methods:

[0030] Clean the data for meaningless stop words;

[0031] Merge, split or replace a word that is mistakenly divided into multiple words to obtain the correct word segmentation result;

[0032] Eliminate junk strings, including idiomatic sentences or sentence patterns;

[0033] Merge synonyms.

[0034] Furthermore, if the definition of a hot word tends to measure the popularity change of a word in the long term, the larger the time parameter T corresponding to the observation group is, the larger the weight corresponding to the observation group is; conversely, if the definition of a hot word tends to measure the popularity change of a word in the short term, the larger the time parameter T corresponding to the observation group is, the smaller the weight corresponding to the observation group is; the time parameter T is calculated in the following way: for an observation group, the actual time distance between each frequency data and the maximum frequency data in the two subsequences is calculated, and the average of all actual time distances is taken as the time parameter T corresponding to the observation group.

[0035] A hot word extraction device of the present invention includes a memory and a processor, wherein the processor is used to execute instructions stored in the memory to implement the hot word extraction method introduced above and achieve the same beneficial effects as the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the hot word extraction method of the present invention;

[0037] Figure 2-1 It is the frequency change graph of candidate word 1;

[0038] Figure 2-2 This is the frequency change graph of candidate word 2;

[0039] Figure 3 It is a structural diagram of the hot word extraction device of the present invention. DETAILED DESCRIPTION

[0040] The basic concept of the present invention is as follows: based on the frequency of candidate words appearing in different time periods, the present invention can obtain the original sequence and sorted sequence of each candidate word, and then analyze the original sequence and sorted sequence of each candidate word from the two aspects of growth capacity and rearrangement loss, and split the original analysis of the growth trend of a frequency sequence into two levels, and then according to the growth capacity and rearrangement loss of the candidate words, as well as the constructed heat model, the heat of each candidate word can be calculated, and finally the real hot words are selected from them according to the heat of each candidate word.

[0041] Among them, the original sequence is obtained by sorting the frequency data in chronological order, and the sorted sequence is obtained by sorting the frequency data according to size relationship; the growth amount can represent the growth rate of the frequency data in the sorted sequence, and the rearrangement loss represents the difference between the original sequence and the sorted sequence.

[0042] Moreover, for the original sequence of a candidate word, when only the original order is changed, the form with the highest popularity should be the form in which all frequencies in the original sequence are monotonically increasing; a hot word with high popularity should have a large growth capacity and a small rearrangement loss, that is, the original sequence is close to an increasing form and the increase is large.

[0043] Based on this, a hot word extraction method and a hot word extraction device of the present invention can be implemented, which are further described in detail below in conjunction with the accompanying drawings and embodiments.

[0044] Method Example:

[0045] An embodiment of a hot word extraction method of the present invention has the following overall process: Figure 1 As shown:

[0046] Step 1: Obtain scientific and technological literature (such as papers, patents, reports, achievements, etc.) within a certain period of time, perform necessary pre-processing on the literature (such as PDF parsing, deduplication processing, text extraction, etc.), and extract the title, keywords, abstract, main content and other texts of the literature as corpus.

[0047] Step 2: Segment the corpus to obtain the segmentation results. In general, a segmentation model can be used to segment the corpus to obtain candidate words. However, considering that the segmentation model based on the training corpus has a weak ability to recognize unregistered words, a segmentation model with a strong ability to recognize unregistered words should be adopted (such as retraining the segmentation model through partially annotated corpus, or using BERT, CRF and other models with strong ability to recognize unregistered words) or introducing new word discovery strategies (such as introducing external dictionaries or knowledge bases).

[0048] Step 3: Clean the word segmentation results obtained in step 2. Since the research object of hot word extraction is words, the accuracy of word segmentation will greatly affect the accuracy of hot word analysis. The word segmentation results need to be cleaned to obtain more accurate word segmentation results. The specific process includes:

[0049] 1) Clean the stop words that have no specific meaning.

[0050] 2) Fine-tune the word segmentation results. Since some proper nouns may be separated due to different word segmentation models, the word segmentation results need to be adjusted through corresponding strategies. For example, words such as gene editing technology "CRISPR / Cas9", new tobacco variety "Zhongyan 100", and chemical detection method "GC-MS" are composed of multiple characters such as Chinese, English, numbers, and special characters. They are easily misclassified by the word segmentation method, and the word segmentation results need to be adjusted, including merging, secondary segmentation, replacement, and other operations.

[0051] 3) Clean up the junk strings in the word segmentation results. Junk strings are often phrases that are used frequently but have no reference value, represented by phrases with a "subject-predicate" structure or a "verb-object" structure, such as "the result is", "research shows", etc. When the corpus is a corpus in a certain field, such as scientific papers, patents, microblogs, news, etc., there will be certain idiomatic sentences and sentence patterns, which will cause such junk strings, and they can be split or removed in a targeted manner.

[0052] 4) Merge synonyms. Synonyms are common in new word discovery, because new words often have not yet formed a unified name. For example, "biochar" and "biomass charcoal", "slim cigarettes" and "slim cigarettes", "tar reduction and harm reduction" and "harm reduction and tar reduction" are all synonyms. When a word is represented by multiple synonyms, its frequency of occurrence will be dispersed, affecting the accuracy of frequency statistics. Furthermore, the similarity between words can be evaluated through word vectors and edit distances, and synonyms can be discovered and merged through cluster analysis.

[0053] Step 4: Use the word segmentation results obtained after step 3 as candidate words, integrate and remove duplicates of all candidate words to form a candidate word table V, where the total number of candidate words is n=|V|.

[0054] Step 5: According to the selected time window (or time period), such as year, month, week, or day, the frequency of each candidate word in the candidate word list is counted in chronological order to obtain a word frequency sequence arranged in chronological order, which is recorded as the original sequence V. i The time sequence here can be arranged from the earliest time to the latest time, or from the latest time to the earliest time. In this embodiment, the original sequence V i is a sequence arranged in time order from far to near. The frequency can be the number of occurrences of a word at that time point, the document frequency, the average number of occurrences, or any monotonic function mapping based on word frequency. The frequency is processed using the plus-one method to avoid zero values ​​and the frequency is normalized.

[0055] According to this method, all candidate words in the candidate word list are processed to obtain the frequency matrix W of all candidate words. As shown in the figure below, the frequency matrix W is the frequency statistics of all candidate words, v i,j represents the frequency data of the i-th word in the j-th time window, and m is the original sequence V i Length.

[0056]

[0057] And according to the original sequence V i The size relationship of each frequency data in the original sequence V i Rearrange, for equal frequencies, in the original order V i Arrange the sequence to obtain a sorted sequence. If the original sequence is arranged in chronological order from far to near, the corresponding sorted sequence needs to be arranged in order from small to large, and the sorted sequence is an increasing sequence; if the original sequence is arranged in chronological order from near to far, the corresponding sorted sequence needs to be arranged in order from large to small, and the sorted sequence is a decreasing sequence. In this embodiment, the sorted sequence uses an increasing sequence V′ i .

[0058] Step 6: Calculate the growth energy of the i-th candidate word. In the process of calculating the growth energy, the present invention designs a frequency change evaluation method based on a sliding window that is insensitive to parameters. The specific calculation process of the growth energy is as follows:

[0059] 1) For the increasing sequence V′ corresponding to a candidate word i , using the observation window to extract subsequences to obtain different observation groups An observation group includes two subsequences, which are extracted by using two windows respectively. The frequency data with the largest frequency value in the increasing sequence is used as the benchmark. The window closer to the benchmark is called the near window, and the window farther from the benchmark is called the far window. The subsequence extracted by the near window is called the first subsequence. The subsequence extracted using the far window is the second subsequence These two subsequences form a group of subsequences, that is, an observation group For the first subsequence and the second subsequence For example, satisfying v′ i,r ≤v′ i,p And v′ i,s ≤v′ i,q (In some cases, the frequency data included in the first subsequence and the second subsequence may overlap).

[0060] In this embodiment, the lengths of the first subsequence and the second subsequence extracted from an observation group are set to be equal, both L, that is, the number of frequency data included in the first subsequence is equal to the number of frequency data included in the second subsequence. The first subsequence is extracted using a window with a window length of L that is closest to the reference, and the second subsequence is extracted using a window with a window length of L that is located before the first subsequence. When L takes different values, different observation groups can be obtained. In this embodiment, within the length range allowed by the frequency data, that is, m<2L, all values ​​of L are taken to obtain different observation groups.

[0061] 2) For different observation groups Set the corresponding weight The configuration of weights is related to the definition of hot words in the application scenario. The time parameter T is used to determine the size of the weight. The meaning of the time parameter T is: for an observation group, the actual time distance between each frequency data in its two subsequences and the benchmark is calculated, and the average of all actual time distances is taken as the time parameter T corresponding to the observation group. If the definition of a hot word tends to change the popularity of the word in the long term, the observation group with a larger T value (i.e., a larger time coverage range) should have a higher weight, and the observation group with a smaller T value (i.e., a smaller time coverage range) should have a smaller weight, so the weight should be positively correlated with the T value; conversely, if the definition of a hot word tends to measure the popularity change of the word in the short term, the weight should be negatively correlated with the T value.

[0062] 3) For an observation group The change of the frequency corresponding to the first subsequence extracted by the near window relative to the frequency corresponding to the second subsequence extracted by the far window is calculated respectively. In this embodiment, the frequency corresponding to the first subsequence extracted by the near window is the average value of all frequency data in the first subsequence, and the frequency corresponding to the second subsequence extracted by the far window is the average value of the frequency data in the second subsequence; and the change is the growth rate of the frequency corresponding to the first subsequence relative to the frequency corresponding to the second subsequence, and the formula is as follows:

[0063]

[0064] 4) Integrate the changes corresponding to each observation group according to the set weight to obtain the growth energy E of the candidate word i In this embodiment, the integration method selected is to perform weighted summation of the changes in each observation group, that is:

[0065]

[0066] Step seven, calculate the rearrangement loss of the i-th candidate word. The main measurement object of the rearrangement loss is the difference between the original form and the incremental form. The measurement method of the rearrangement loss is inspired by the edit distance, which is the number of operations required to adjust two strings to be consistent. In the process of morphological transformation, there is no addition and deletion, so there is only an exchange operation (the exchange operation of each frequency data in the original sequence). The loss of the exchange considers two factors. One is the interval distance between the two frequency data to be exchanged, and the exchange of two frequency data with a long interval should cause a large loss. The other is the frequency difference between the two frequency data to be exchanged, and the exchange of two frequency data with a large frequency difference should cause a large loss. The rearrangement loss is the loss caused by converting the frequency data in the original form into an incremental form through exchange. The specific process is as follows:

[0067] 1) Get the increasing sequence V′ corresponding to the i-th candidate word i .

[0068] 2) For the increasing sequence V′ i The kth frequency data v′ in i,k , for the increasing sequence V′ i Traverse and find the frequency data v corresponding to the intermediate sequence obtained after the last frequency data exchange i,j , and the frequency data v i,j The first position after k-1 satisfies v i,j =v′ i,k , calculate v i,j and v′ i,k The position interval D between them:

[0069]

[0070] 3) For the intermediate sequence obtained after the last frequency data exchange, replace v in step 2) i,j The frequency data v at the kth position corresponding to it i,k Interchange them to obtain the intermediate sequence after the frequency data is interchanged, and calculate the frequency difference S between the two:

[0071]

[0072] 4) For the increasing sequence V′ i For each frequency data in the frequency data, repeat steps 2) and 3) in order from small to large until V i and V′ i Same, now calculate the re-ranking loss R of the i-th candidate word i :

[0073]

[0074] Step 8: The growth amount of the i-th candidate word can be E i and the rearrangement loss R i Substitute into the constructed heat model and calculate the heat H of the i-th candidate word i The relationship reflected by the heat model is: heat H i and growth capacity E i Positive correlation, heat H i and the rearrangement loss R i Negative correlation.

[0075] Further analysis of the magnitude of the growth energy and rearrangement loss shows that the theoretical minimum value of the growth energy E under the condition of equal weight in the observation window is not less than -m / 2, and the maximum value is greatly affected by frequency fluctuations and may be infinite. The minimum value of the magnitude of the rearrangement loss is 0, and the theoretical maximum value does not exceed m-1, where m is the length of the original sequence.

[0076] When combining the growth capacity and the rearrangement loss, three factors should be considered comprehensively: ① The contribution of E and R to the heat should be in the same order of magnitude; ② When the rearrangement loss is 0, H should be equivalent to E; ③ The rearrangement loss does not dominate the heat calculation (that is, the heat of the candidate word can be lower due to excessive rearrangement loss, but it cannot be unilaterally higher due to too small rearrangement loss).

[0077] In this embodiment, the E i The maximum value is set to avoid the influence of extreme values ​​on the statistical results. The plus-one method is used to avoid the denominator being 0. At the same time, it is avoided that the rearrangement loss is too small to dominate. The final heat model is set as follows:

[0078]

[0079] Step 9: traverse the frequency matrix W, and use the original sequences of the remaining candidate words in the frequency matrix W to calculate their corresponding heat H according to the method of steps 6 to 8 i After obtaining the heat of each candidate word, according to the heat threshold, the candidate words with heat higher than the heat threshold are screened out as the extracted hot words.

[0080] The effectiveness of the present invention is verified below with reference to a specific example.

[0081] The original sequence of the frequency data of candidate word 1 and candidate word 2 counted from the corpus is as follows:

[0082] Original sequence of candidate word 1 = [3 2 3 2 5 3 4 6 9 10]

[0083] Original sequence of candidate word 2 = [5 9 6 8 4 6 3 5 0 4]

[0084] The frequency changes of candidate word 1 and candidate word 2 are as follows: Figure 2-1 and Figure 2-2 As shown. Judging from the frequency changes of the two candidate words, candidate word 1 shows an accelerating upward trend, and candidate word 2 shows a slightly downward trend. Therefore, in theory, the popularity of candidate word 1 should be higher than that of candidate word 2. As a comparative verification, the popularity of the candidate words is statistically analyzed using comparison method one, comparison method two, and the method described in the present invention. Comparison method one uses a method for automatic extraction of hot words for agricultural public opinion disclosed in the Chinese invention patent application with authorization announcement number CN107967299B. Comparison method two uses the method disclosed in "Research on Short Text Hot Topic Detection Based on Word Frequency Mean Fluctuation and Probabilistic Language Model" published in 2019.

[0085] 1. Comparison method 1.

[0086] First, the basic weights of candidate 1 and candidate 2 are calculated respectively, and the calculation formula is as follows:

[0087] B=lg(1+lg(1+lg(t f +1)))

[0088] Among them, t f is the frequency of a single time window.

[0089] Then, through calculation, the basic weights of candidate word 1 and candidate word 2 are obtained as follows:

[0090] The basic weight of candidate word 1 is B = [0.63 0.55 0.63 0.55 0.71 0.63 0.67 0.73 0.79 0.80]

[0091] The basic weight of candidate word 2 is B = [0.71 0.79 0.73 0.77 0.67 0.73 0.63 0.71 0.000.67]

[0092] Next, calculate the overall volatility V, long-term change L, short-term change S, and volatility weight F respectively. The calculation methods are as follows:

[0093]

[0094]

[0095]

[0096] F=V*0.4+L*0.4+S*0.2

[0097] The calculation results for candidate word 1 and candidate word 2 are as follows:

[0098] The fluctuation weight of candidate word 1 = 0.74

[0099] The fluctuation weight of candidate word 2 = 1.02

[0100] Finally, the heat weight H is obtained, and its calculation formula is as follows:

[0101] H=B*0.5+F*0.5

[0102] The heat of candidate word 1 and candidate word 2 calculated in this way are as follows:

[0103] The heat of candidate word 1 = 0.77

[0104] The heat of candidate word 2 = 0.85

[0105] That is, the conclusion drawn through comparison method 1 is that the popularity of candidate word 2 is higher than that of candidate word 1.

[0106] 2. Comparison method 2.

[0107] First, for candidate word 1 and candidate word 2, the frequency fluctuation values ​​of their adjacent time windows are calculated respectively, and the positive fluctuation values ​​are added to obtain their fluctuation coefficients. The words with high fluctuation coefficients are regarded as significant fluctuation words, and then high-frequency hot words are obtained by sorting the word frequencies. The calculation formula of the fluctuation coefficient is as follows:

[0108] δ=∑Kw i ·t n / V(Kw i ·t n >0)

[0109] Among them, Kw i ·t n For the same candidate word in t n +1 with t n The word frequency difference in the time window, V is the total number of times the candidate word appears.

[0110] The fluctuation coefficients of candidate word 1 and candidate word 2 are calculated as follows:

[0111] The fluctuation coefficient of candidate word 1 = 0.23

[0112] The fluctuation coefficient of candidate word 2 = 0.28

[0113] It can be concluded that the volatility coefficient of candidate word 2 is higher than that of candidate word 1, and it is more likely to be selected as a hot word.

[0114] 3. The method of the present invention.

[0115] First, calculate the frequency sequence of candidate word 1 and candidate word 2 respectively after adding one, normalizing, and retaining two decimal places, that is, the original sequence:

[0116] V1=[0.07 0.05 0.07 0.05 0.11 0.07 0.09 0.12 0.18 0.19]

[0117] V2=[0.10 0.17 0.12 0.15 0.08 0.12 0.07 0.10 0.02 0.08]

[0118] After arranging in ascending order, we get an ascending sequence:

[0119] V′1=[0.05 0.05 0.07 0.07 0.07 0.09 0.11 0.12 0.18 0.19]

[0120] V′2=[0.02 0.07 0.08 0.08 0.10 0.10 0.12 0.12 0.15 0.17]

[0121] Then, the growth capacity and rearrangement loss of candidate word 1 and candidate word 2 are calculated respectively, so as to finally calculate the popularity of the two candidate words. The details are as follows.

[0122] 1) Calculate the growth volume.

[0123] ① Select different near and far windows to extract different observation groups. Extract different observation groups with L ranging from 1 to 5, and get the following results:

[0124]

[0125]

[0126] ② Without setting any weights for different observation groups, the corresponding growth volume can be calculated as:

[0127] E1=0.06+0.61+0.81+1.00+1.23=3.71

[0128] E2=0.13+0.33+0.38+0.56+0.89=2.29

[0129] 2) Calculate the rearrangement loss.

[0130] ①For V1 and V′1:

[0131] V1=[0.07 0.05 0.07 0.05 0.11 0.07 0.09 0.12 0.18 0.19]

[0132] V′1=[0.05 0.05 0.07 0.07 0.07 0.09 0.11 0.12 0.18 0.19]

[0133] Perform the following swap operations:

[0134] Operation 1: For v′ 1,1 , the corresponding one in V1 is v 1,2 , swap v 1,2 With v 1,1 , the loss value is 0.02*1=0.02, and the following intermediate sequence is obtained:

[0135] [v 1,2 v 1,1 v 1,3 v 1,4 v 1,5 v 1,6 v 1,7 v 1,8 v 1,9 v 1,10 ]

[0136] Operation 2: For v′ 1,2 , the corresponding one in V1 is v 1,4 , swap v 1,4 With v 1,1 , the loss value is 0.02*2=0.04, and the following intermediate sequence is obtained:

[0137] [v 1,2 v 1,4 v 1,3 v 1,1 v 1,5 v 1,6 v 1,7 v 1,8 v 1,9 v 1,10 ]

[0138] Operation 3: For v′ 1,3 , the corresponding one in V1 is v 1,1 , swap v 1,1 With v 1,3 , the loss value is 0, and the following intermediate sequence is obtained:

[0139] [v 1,2 v 1,4 v 1,1 v 1,3 v 1,5 v 1,6 v 1,7 v 1,8 v1,9 v 1,10 ]

[0140] Operation 4: For v′ 1,4 , the corresponding one in V1 is v 1,3 , its position remains unchanged and the loss value is 0.

[0141] [v 1,2 v 1,4 v 1,1 v 1,3 v 1,5 v 1,6 v 1,7 v 1,8 v 1,9 v 1,10 ]

[0142] Operation 5: For v′ 1,5 , the corresponding one in V1 is v 1,6 , swap v 1,6 With v 1,5 , the loss value is 0.04*1=0.04, and the following intermediate sequence is obtained:

[0143] [v 1,2 v 1,4 v 1,1 v 1,3 v 1,6 v 1,5 v 1,7 v 1,8 v 1,9 v 1,10 ]

[0144] Operation 6: For v′ 1,6 , the corresponding one in V1 is v 1,7 , swap v 1,7 With v 1,5 , the loss value is 0.02*1=0.02, and the following intermediate sequence is obtained:

[0145] [v 1,2 v 1,4 v 1,1 v 1,3 v 1,6 v 1,7 v 1,5 v 1,8 v 1,9 v 1,10 ]

[0146] Operation 7: For v′ 1,7 , the corresponding one in V1 is v 1,5 , its position remains unchanged and the loss value is 0.

[0147] Operation 8: For v′ 1,8 , the corresponding one in V1 is v 1,8 , its position remains unchanged and the loss value is 0.

[0148] Operation 9: For v′ 1,9 , the corresponding one in V1 is v 1,9 , its position remains unchanged and the loss value is 0.

[0149] Operation 10: For v′ 1,10 , the corresponding one in V1 is v 1,10 , its position remains unchanged and the loss value is 0.

[0150] The final transformed sequence is the same as V′1, R1=0.2+0.04+0.04+0.02=0.12.

[0151] ②For V2 and V′2:

[0152] V2=[0.10 0.17 0.12 0.15 0.08 0.12 0.07 0.10 0.02 0.08]

[0153] V′2=[0.02 0.07 0.08 0.08 0.10 0.10 0.12 0.12 0.15 0.17]

[0154] Perform the following swap operations:

[0155] Operation 1: For v′ 2,1 , the corresponding one in V2 is v 2,9 , swap v 2,9 With v 2,1 , the loss value is 0.08*8=0.64, and the following intermediate sequence is obtained:

[0156] [v 2,9 v 2,2 v 2,3 v 2,4 v 2,5 v 2,6 v 2,7 v 2,8 v 2,1 v 2,10 ]

[0157] Operation 2: For v′ 2,2 , the corresponding one in V2 is v 2,7 , swap v 2,7 With v 2,2 , the loss value is 0.10*5=0.50, and the following intermediate sequence is obtained:

[0158] [v 2,9 v 2,7 v 2,3 v 2,4 v 2,5 v 2,6 v 2,2 v 2,8 v 2,1 v 2,10 ]

[0159] Operation 3: For v′ 2,3 , the corresponding one in V2 is v 2,5 , swap v 2,5 With v 2,3 , the loss value is 0.04*2=0.08, and the following intermediate sequence is obtained:

[0160] [v 2,9 v 2,7 v 2,5 v 2,4 v 2,3 v 2,6 v 2,2 v 2,8 v 2,1 v 2,10 ]

[0161] Operation 4: For v′ 2,4 , the corresponding one in V2 is v 2,10 , swap v 2,10 With v 2,4 , the loss value is 0.07*6=0.42, and the following intermediate sequence is obtained:

[0162] [v 2,9 v 2,7 v 2,5 v 2,10 v 2,3 v 2,6 v 2,2 v 2,8 v 2,1 v 2,4 ]

[0163] Operation 5: For v′ 2,5 , the corresponding one in V2 is v 2,8 , swap v 2,8 With v 2,3 , the loss value is 0.02*3=0.06, and the following intermediate sequence is obtained:

[0164] [v 2,9 v 2,7 v 2,5 v 2,10 v 2,8 v2,6 v 2,2 v 2,3 v 2,1 v 2,4 ]

[0165] Operation 6: For v′ 2,6 , the corresponding one in V2 is v 2,1 , swap v 2,1 With v 2,6 , the loss value is 0.02*3=0.06, and the following intermediate sequence is obtained:

[0166] [v 2,9 v 2,7 v 2,5 v 2,10 v 2,8 v 2,1 v 2,2 v 2,3 v 2,6 v 2,4 ]

[0167] Operation 7: For v′ 2,7 , the corresponding one in V2 is v 2,3 , swap v 2,3 With v 2,2 , the loss value is 0.05*1=0.05, and the following intermediate sequence is obtained:

[0168] [v 2,9 v 2,7 v 2,5 v 2,10 v 2,8 v 2,1 v 2,3 v 2,2 v 2,6 v 2,4 ]

[0169] Operation 8: For v′ 2,8 , the corresponding one in V2 is v 2,6 , swap v 2,6 With v 2,2 , the loss value is 0.05*1=0.05, and the following intermediate sequence is obtained:

[0170] [v 2,9 v 2,7 v 2,5 v 2,10 v 2,8 v 2,1 v 2,3 v 2,6 v 2,2 v 2,4 ]

[0171] Operation 9: For v′ 2,9 , the corresponding one in V2 is v 2,4 , swap v 2,4 With v 2,2 , the loss value is 0.02*1=0.02, and the following intermediate sequence is obtained:

[0172] [v 2,9 v 2,7 v 2,5 v 2,10 v 2,8 v 2,1 v 2,3 v 2,6 v 2,4 v 2,2 ]

[0173] Operation 10: For v′ 2,10 , the corresponding one in V2 is v 2,2 , its position remains unchanged and the loss value is 0.

[0174] The final transformed sequence is the same as V′2, R2=0.64+0.50+0.08+0.42+0.06+0.06+0.05+0.05+0.02=1.88.

[0175] 3) Calculate heat.

[0176] Through calculation, the heat of candidate word 1 and candidate word 2 is as follows:

[0177]

[0178]

[0179] It can be concluded that the popularity of candidate word 1 is higher than that of candidate word 2.

[0180] For comparison, the following table lists the results of the three methods.

[0181] Table 1

[0182] Method of the present invention Comparison method 1 Comparison method 2 Candidate word 1 heat 3.37 0.77 0.23 Candidate word 2 heat 0.80 0.85 0.28

[0183] Since it can be seen from the frequency change trend that the popularity of candidate word 1 should be significantly higher than that of candidate word 2, it is verified that the hot word extraction method of the present invention has advantages in the following aspects: the present invention can effectively avoid the influence of noise fluctuations and has a higher accuracy rate.

[0184] In summary, the method of the present invention splits the traditional analysis of the growth trend of a frequency series into two levels: growth capacity and rearrangement loss. It effectively avoids the influence of noise fluctuations while retaining the original information, and the accuracy of hot word extraction is relatively high. At the same time, compared with the comparative method, the model has strong universality and usability, does not rely on any data set, and is insensitive to parameters.

[0185] A frequency data interchange method is provided in step 7 of this embodiment. As another implementation method, all interchange operations can be traversed through a search strategy such as breadth-first search (BFS) or depth-first search (DFS), and the path with the smallest rearrangement loss is selected as the frequency data interchange method. From an empirical perspective, the frequency data interchange method of the present invention is equivalent to the interchange method with the smallest rearrangement loss.

[0186] In step 6 of this embodiment, the growth rate is used to characterize the difference between two subsequences in an observation group. As other implementations, a difference, a ratio, etc. can also be selected as the difference between two subsequences.

[0187] In this embodiment, the original sequence is arranged in order from far to near in time, and the sorted sequence (in this case, an ascending sequence) is arranged in order from small to large. As another embodiment, the original sequence can also be arranged in order from near to far in time, and the sorted sequence (in this case, a descending sequence) is arranged in order from large to small. The principles of the subsequent analysis and processing of the two sorting methods are the same.

[0188] Device Example:

[0189] An embodiment of a hot word extraction device of the present invention is as follows: Figure 3 As shown, it includes a memory, a processor and an internal bus, and the processor and the memory communicate and exchange data with each other through the internal bus. The memory includes at least one software function module stored in the memory, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory to implement a hot word extraction method introduced in the method embodiment of the present invention.

[0190] The processor may be a microprocessor MCU, a programmable logic device FPGA or other processing device. The memory may be various memories that store information using electrical energy, such as RAM, ROM, etc.; it may also be various memories that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, magnetic bubble memories, USB flash drives, etc.; it may also be various memories that store information using optical methods, such as CDs, DVDs, etc.; of course, it may also be other types of memories, such as quantum memories, graphene memories, etc.

Claims

1. A hot word extraction method, characterized in that: The steps include: 1) Obtain all candidate words and count the frequency data of each candidate word in different time periods; 2) For the i-th candidate word, sort the frequency data in chronological order to obtain the original sequence V i , sort the frequency data according to the frequency relationship to obtain the sorting sequence V′ i ; 3) The following method is used to calculate the growth energy E of the i-th candidate word: i : Extract the sorted sequence V′ of the i-th candidate word i different observation groups, one observation group includes two different subsequences, namely a first subsequence and a second subsequence, the maximum value of the frequency data in the first subsequence is greater than or equal to the maximum value of the frequency data in the second subsequence, and the minimum value of the frequency data in the first subsequence is greater than or equal to the minimum value of the frequency data in the second subsequence; Calculate the difference between the first subsequence and the second subsequence in an observation group; According to the difference between the two subsequences in each observation group, the growth energy E of the i-th candidate word is calculated. i ; 4) Calculate the rearrangement loss R of the i-th candidate word using the following method: i : The i-th candidate word is taken from the original sequence V i Transform to sorted sequence V′ i During the process, the position interval and frequency difference between two frequency data to be exchanged are determined each time the frequency data is exchanged; According to the position interval and frequency difference corresponding to each frequency data exchange, the reordering loss R of the i-th candidate word is calculated i : 5) The growth amount of the i-th candidate word can be E i and the rearrangement loss R i Substitute it into the constructed heat model and calculate the heat H of the i-th candidate word i The relationship reflected by the heat model is: the growth energy E i With heat H i Positively correlated, rearrangement loss R i With heat H i Negative correlation; 6) Calculate the heat of each candidate word according to steps 2) to 5), and select candidate words whose heat is greater than the heat threshold as extracted hot words.

2. The hot word extraction method according to claim 1, characterized in that: In step 3), the difference between the first subsequence and the second subsequence in the one observation group is the growth rate of the frequency corresponding to the first subsequence compared to the frequency corresponding to the second subsequence, and the frequency corresponding to the first subsequence is the average value of all frequency data in the first subsequence, and the frequency corresponding to the second subsequence is the average value of all frequency data in the second subsequence.

3. The hot word extraction method according to claim 1 or 2, characterized in that: In step 3), the growth energy E of the i-th candidate word is calculated based on the difference between the two subsequences in each observation group. i The method used is: weighted sum the differences between the two subsequences of each observation group to obtain the growth energy E of the i-th candidate word i .

4. The hot word extraction method according to claim 1, characterized in that: In step 4), the rearrangement loss R of the i-th candidate word is calculated based on the position interval and frequency difference corresponding to each frequency data exchange. i The method is as follows: multiply the position interval and frequency difference corresponding to a frequency data exchange to obtain the product result corresponding to a frequency data exchange; add the product results corresponding to each frequency data exchange to obtain the reordering loss R of the i-th candidate word i .

5. The hot word extraction method according to claim 1 or 4, characterized in that: In step 4), in order to remove the i-th candidate word from the original sequence V i Transform to sorted sequence V′ i , the frequency data exchange method used is: From the sorted sequence V′ i The frequency data at the first position in the original sequence V is interchanged in the following manner. i With sorted sequence V′ i Same: For sorted sequence V′ i The kth frequency data v′ in i,k , find the first intermediate sequence that satisfies v starting from the kth position after the last frequency data exchange i,j =v′ i,k Frequency data v i,j , and replace the frequency data v in the intermediate sequence obtained after the last frequency data exchange i,j With frequency data v i,k The positions of the two frequencies are swapped to obtain the intermediate sequence after the frequency data is swapped.

6. The hot word extraction method according to claim 1, characterized in that: In step 5), the heat model is: Where m is the original sequence V i Length.

7. The hot word extraction method according to claim 1, characterized in that: The method for obtaining candidate words in step 1) is: obtaining various documents and extracting corpus from the documents; performing word segmentation processing on the corpus to obtain word segmentation results; and determining candidate words based on the word segmentation results.

8. The hot word extraction method according to claim 7, characterized in that: The method for determining candidate words according to the word segmentation results includes: performing data cleaning on the word segmentation results to obtain candidate words; the data cleaning method includes at least one of the following methods: Clean the data for meaningless stop words; Merge, split or replace a word that is mistakenly divided into multiple words to obtain the correct word segmentation result; Eliminate junk strings, including idiomatic sentences or sentence patterns; Merge synonyms.

9. The hot word extraction method according to claim 3, characterized in that: If the definition of a hot word tends to measure the popularity change of a word in the long term, the larger the time parameter T corresponding to the observation group is, the larger the weight corresponding to the observation group is; conversely, if the definition of a hot word tends to measure the popularity change of a word in the short term, the larger the time parameter T corresponding to the observation group is, the smaller the weight corresponding to the observation group is; the time parameter T is calculated in the following way: for an observation group, the actual time distance between each frequency data and the maximum frequency data in the two subsequences is calculated, and the average of all actual time distances is taken as the time parameter T corresponding to the observation group.

10. A hot word extraction device, characterized in that: It comprises a memory and a processor, wherein the processor is used to execute instructions stored in the memory to implement the hot word extraction method as described in any one of claims 1 to 9.

Images