A method, device, electronic device and storage medium for determining hot words

Abstract

The embodiment of the present invention is applicable to the field of data processing technology, and provides a hot word determination method, device, electronic device and storage medium, wherein the hot word determination method includes: performing word segmentation processing on a text to be processed to obtain at least two word segments; performing part-of-speech tagging on each of the at least two word segments; combining the word segments in the at least two word segments after the part-of-speech tagging to obtain at least one combined word; the combined word is composed of the word segments in the at least two word segments that are adjacent in position in the text to be processed; determining the heat value of each combined word in the at least one combined word; the heat value represents the frequency of occurrence of the corresponding combined word in the text to be processed; determining the hot words in the text to be processed based on the heat value; the hot word represents the combined word whose heat value is greater than the set value.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, apparatus, electronic device, and storage medium for determining hot words. Background Art

[0002] In the e-commerce field, abnormal behaviors such as "taking advantage of loopholes" will cause double losses to merchants and platforms. By extracting hot words from text information on the Internet, business personnel can provide decision-making basis for optimizing marketing strategies and performing risk control interception by analyzing the hot words, and timely discover abnormal behaviors such as "taking advantage of loopholes". When extracting hot words in related technologies, the text is simply segmented only according to the part of speech, resulting in low accuracy of the extracted hot words. Summary of the Invention

[0003] To solve the above problems, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for determining hot words to at least solve the problem of low accuracy of the hot words extracted by related technologies.

[0004] The technical solution of the present invention is implemented as follows:

[0005] In a first aspect, an embodiment of the present invention provides a method for determining hot words, the method including:

[0006] Performing word segmentation on the text to be processed to obtain at least two segmented words;

[0007] Performing part-of-speech tagging on each of the at least two segmented words;

[0008] Combining the segmented words in the at least two segmented words with part-of-speech tags to obtain at least one combined word; the combined word is composed of the segmented words that are adjacent in position in the text to be processed;

[0009] Determining the heat value of each combined word in the at least one combined word; the heat value represents the frequency of occurrence of the corresponding combined word in the text to be processed;

[0010] Determining hot words in the text to be processed based on the heat value; the hot words represent combined words with a heat value greater than a set value.

[0011] In an embodiment, the combining the segmented words in the at least two segmented words with part-of-speech tags to obtain at least one combined word includes:

[0012] Combining the segmented words in the at least two segmented words with part-of-speech tags based on a first set rule to obtain at least one first combined word;

[0013] Based on the second setting rule, the first combination word in the at least one first combination word and the participle in the at least two participles are combined to obtain at least one second combination word; each second combination word in the at least one second combination word includes a first combination word.

[0014] In one embodiment, each of the at least one combination word includes at least one numeral.

[0015] In one embodiment, determining the popularity value of each combination word in the at least one combination word includes:

[0016] Clustering each of the at least one combination of words to obtain at least two combination of words sets; each combination of words sets in the at least two combination of words sets corresponds to a different risk level;

[0017] Determine the heat value of the combination word in the combination word set with the highest risk in the at least two combination word sets.

[0018] In one embodiment, clustering each of the at least one combined word to obtain at least two combined word sets includes:

[0019] Vectorize each of the at least one combined word to obtain at least one vector;

[0020] Determine the distance value between each vector in the at least one vector and at least two set cluster centers; each cluster center in the at least two set cluster centers corresponds to a combination word set;

[0021] A combination word set corresponding to each combination word in the at least one combination word is determined based on the distance value.

[0022] In one embodiment, determining the popularity value of each combination word in the at least one combination word includes:

[0023] Determine the value of a first parameter and a value of a second parameter of each of the at least one combination of words; the first parameter represents the number of times the corresponding combination of words appears in the text to be processed; the second parameter represents the distribution of the corresponding combination of words in the text to be processed;

[0024] The popularity value of the corresponding combination word is determined based on the value of the first parameter and the value of the second parameter.

[0025] In one embodiment, the method further comprises:

[0026] Sorting the at least one combination of words based on the popularity value to obtain a sorting result;

[0027] The sorting result is displayed and outputted.

[0028] In a second aspect, an embodiment of the present invention provides a hot word determination device, the device comprising:

[0029] A word segmentation module is used to perform word segmentation on the text to be processed to obtain at least two word segments;

[0030] A tagging module, used for performing part-of-speech tagging on each of the at least two participles;

[0031] A combining module, used for combining the at least two participles with parts of speech marked thereon to obtain at least one combined word; the combined word is composed of the at least two participles that are adjacent in position in the text to be processed;

[0032] A first determination module is used to determine the popularity value of each combination word in the at least one combination word; the popularity value represents the frequency of occurrence of the corresponding combination word in the text to be processed;

[0033] The second determination module is used to determine the hot words in the text to be processed based on the heat value; the hot words represent the combination words whose heat value is greater than a set value.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the steps of the hot word determination method provided in the first aspect of the embodiment of the present invention.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, including: the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the hot word determination method provided in the first aspect of the embodiment of the present invention are implemented.

[0036] The embodiment of the present invention performs word segmentation processing on the text to be processed to obtain at least two word segmentations, performs part-of-speech tagging on each of the at least two word segmentations, and combines the word segmentations in the at least two word segmentations that are marked with part-of-speech to obtain at least one combined word, wherein the combined word is composed of the word segmentations in the at least two word segmentations that are adjacent in position in the text to be processed. The heat value of each combined word in the at least one combined word is determined, and the hot words in the text to be processed are determined based on the heat value. The heat value represents the frequency of occurrence of the corresponding combined word in the text to be processed, and the hot word represents a combined word whose heat value is greater than a set value. The embodiment of the present invention prevents the key information from being disassembled during the word segmentation process by combining the word segmentations in the at least two word segmentations that are marked with part-of-speech, and the extracted hot words are more accurate, which is convenient for business personnel to adjust marketing strategies according to the hot words. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of an implementation flow of a hot word determination method provided by an embodiment of the present invention;

[0038] Figure 2 It is a schematic diagram of the implementation flow of another hot word determination method provided by an embodiment of the present invention;

[0039] Figure 3 It is a schematic diagram of the implementation flow of another hot word determination method provided by an embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of the implementation flow of another hot word determination method provided by an embodiment of the present invention;

[0041] Figure 5 It is a schematic diagram of the implementation flow of another hot word determination method provided by an embodiment of the present invention;

[0042] Figure 6 It is a schematic diagram of the implementation flow of another hot word determination method provided by an embodiment of the present invention;

[0043] Figure 7 It is a schematic diagram of a hot word determination process provided by an application embodiment of the present invention;

[0044] Figure 8 is a schematic diagram of a hot word determination device provided by an embodiment of the present invention;

[0045] Fig. 9 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] As major e-commerce companies continue to increase their investment in marketing activities, there are more and more "wool" such as red envelope cashback, coupons, and free coupons, and forums such as "wool-pulling" and "discount websites" continue to emerge. Public opinion hot word extraction technology uses text information generated on the Internet to analyze the risk model of marketing strategies, dynamically and intuitively display hot words generated over a period of time, and provide a scientific method for intelligent marketing risk control. By analyzing public opinion hot words, it provides decision-making basis for business personnel to optimize marketing strategies and conduct risk control interception, timely discover abnormal behaviors such as "wool-pulling", reduce corporate losses, and improve marketing effectiveness.

[0048] Public opinion hot words refer to phrases that are hotly discussed online, have a wide range of influence, and appear frequently online during a specific period of time after an event occurs. In the field of e-commerce, public opinion hot words include key words such as red envelope cashback, coupons, and free coupons, such as "10 off for purchases over 199 yuan", "8.8 yuan red envelope", "199-120 coupons", "30 buys 300", etc.

[0049] The research and development focus of e-commerce marketing risk control public opinion hot words is different from that of other network public opinion hot words. Currently, most of the existing public opinion analysis methods are sentiment analysis, which judges people's emotional tendencies and emotional changes towards public events. Most of the methods for extracting public opinion hot words are used for public opinion monitoring and solving problems such as network public opinion crisis public relations. When it comes to e-commerce marketing risk control scenarios, the relevant technologies only simply segment the text according to the part of speech when extracting hot words, resulting in low accuracy of the extracted hot words. The hot words extracted by the relevant technologies do not match the e-commerce marketing risk control scenarios and have no reference value for marketing risk control.

[0050] In view of the shortcomings of the above-mentioned related technologies, an embodiment of the present invention provides a method for determining hot words, which can at least improve the accuracy of the extracted hot words. In order to illustrate the technical solution of the present invention, a specific embodiment is used for description below.

[0051] Figure 1 1 is a schematic diagram of an implementation flow of a method for determining a hot word provided by an embodiment of the present invention. The execution subject of the method for determining a hot word may be an electronic device, and the electronic device may be a desktop computer, a laptop computer, a server, etc. Figure 1 , the hot word determination methods include:

[0052] S101, performing word segmentation processing on the text to be processed to obtain at least two word segments.

[0053] Here, the text to be processed is text related to the e-commerce marketing risk control scenario. For example, the text to be processed can be comments from netizens on online forums or chat texts in communication software. For example, assuming that there is a WeChat group specifically for online shopping discounts, the text to be processed can be the chat records generated in the WeChat group over a period of time. The text to be processed can also be comments from netizens in the comment area corresponding to the products in the shopping software.

[0054] The text to be processed is segmented, mainly Chinese word segmentation, which is the basis of Chinese text classification. Unlike English, there are no fixed separators between words in Chinese characters, and the boundaries are fuzzy. It is difficult for computers to accurately identify the meaning of Chinese words and complete text processing tasks. Therefore, the first step of hot word extraction based on Chinese text datasets is Chinese word segmentation.

[0055] Chinese word segmentation algorithms are divided into three categories: mechanical matching-based word segmentation algorithms, statistics-based word segmentation algorithms, and understanding-based word segmentation algorithms. General Chinese word segmentation tools include: jieba, Hanlp, SnowNLP (MIT), pynlpir (Big Data Search and Mining Laboratory (Beijing Engineering Research Center for Massive Language Information Processing and Cloud Computing Application)), thulac (Natural Language Processing and Social Humanities Computing Laboratory of Tsinghua University), etc.

[0056] In the embodiments of the present invention, jieba Chinese word segmentation can be adopted. Among them, it is necessary to construct a user-defined word library, which is used to define words or phrases that may appear in some scenarios to prevent a word from being segmented into multiple words during the word segmentation process.

[0057] In practical applications, if the text to be classified is in Chinese, the jieba word segmentation tool can be used to segment the text to be classified. Jieba is a Python Chinese word segmentation component. For example, if a sentence in the text to be classified is "Go to my home for dinner tomorrow", the corresponding word segmentation set is "tomorrow, go, my home, dinner". If the text to be classified is in English, since each word in an English sentence is separated by a space, the English sentence can be segmented according to the space. For example, if the English sentence is "Tom chase Jerry", the corresponding word segmentation set is "Tom, chase, Jerry".

[0058] When performing word segmentation, it is also necessary to remove stop words and punctuation marks in the word segmentation. Stop words are words that do not contain much information, such as some high-frequency pronouns, conjunctions, prepositions, etc. in the text to be classified. Stop words and punctuation marks are meaningless for text classification, so they need to be removed. In practical applications, English stop words include "the", "of", "for", "with", "to", etc.; Chinese stop words include "ah", "le", "and", "therefore", etc.

[0059] S102, perform part-of-speech tagging on each of the at least two word segments.

[0060] Part-of-speech tagging (POS tagging) refers to the process of marking a suitable part of speech for each word in a sentence, that is, determining whether each word is a noun, verb, adverb, adjective, preposition or other part of speech. The jieba part-of-speech tagging is adopted in the present invention.

[0061] For example, in the sentence "I want to eat mangoes", among them, the part of speech of the word segment "I" is a noun, the part of speech of the word segment "eat" is a verb, and the part of speech of the word segment "mangoes" is a noun.

[0062] S103, combining the at least two participles with marked parts of speech to obtain at least one combined word; the combined word is composed of the at least two participles that are adjacent in position in the text to be processed.

[0063] Traditional word segmentation technology only performs simple segmentation based on part of speech, but in marketing scenarios, there is usually a special type of part of speech, such as the frequent appearance of "199-10" and other number + symbol + number combinations when receiving coupons, and the "8.8 yuan red envelope" number + quantifier + noun combination, which means that the part of speech needs to be specially processed. The method used in the embodiment of the present invention is to perform a secondary combination of the parts of speech after marking the parts of speech to identify the special parts of speech that exist in the e-commerce marketing scenario.

[0064] Here, the compound word is composed of adjacent segment words in the text to be processed. For example, the text to be processed includes the short text "Go to the group to grab an 8.8 yuan red envelope", in which the compound word "8.8 yuan red envelope" is composed of adjacent segment words in the text to be processed.

[0065] Adjacent participles can be combined according to pre-set rules, and the rules can specify the number and part of speech of the participles in the combined word.

[0066] refer to Figure 2 In one embodiment, the combining of the at least two participles with marked parts of speech to obtain at least one combined word includes:

[0067] S201, combining the at least two participles marked with parts of speech based on a first setting rule to obtain at least one first combined word.

[0068] Here, the first setting rule specifies the part of speech of the participle in the combination word, for example, the first setting rule is: numeral + verb, numeral + numeral, numeral + English, numeral + quantifier, numeral + noun, numeral + other proper noun, numeral + modal particle + morpheme, numeral + name, numeral + non-morpheme word + numeral, numeral + non-morpheme word + non-morpheme word. According to the above rules, the participles in at least two participles are combined to obtain the first combination word.

[0069] S202, combining a first combination word in the at least one first combination word and a participle in the at least two participles based on a second setting rule to obtain at least one second combination word; each second combination word in the at least one second combination word includes a first combination word.

[0070] Here, the second setting rule stipulates that the second combined word must include a first combined word. For example, the second setting rule is: first combined word + modal particle + quantifier, first combined word + conjunction + modal particle, first combined word + conjunction + noun, first combined word + conjunction + English word, first combined word + conjunction + numeral, first combined word + conjunction + nominal morpheme. Combine the first combined words in at least one first combined word and the participles in at least two participles according to the above rules to obtain at least one second combined word.

[0071] In an embodiment of the present invention, at least one combined word includes a first combined word and a second combined word, and both the first combined word and the second combined word include at least one numeral.

[0072] In one embodiment, each combined word in the at least one combined word includes at least one numeral.

[0073] In the e-commerce marketing scenario, abnormal behaviors such as "taking advantage of loopholes" are usually associated with red envelope cashback, coupons, free order vouchers, etc., and red envelope cashback, coupons, free order vouchers, etc. are related to amount numbers. Therefore, words related to numbers in the e-commerce marketing scenario are relatively crucial, and the combined word must include a numeral. For example, the combined word is "8.8 yuan red envelope", which is composed of the participles "8.8", "yuan", and "red envelope".

[0074] For the e-commerce marketing risk control scenario, combining participles can prevent key information from being disassembled during the participle process, resulting in the inability to identify key hot words.

[0075] S104, determine the heat value of each combined word in the at least one combined word; the heat value represents the frequency of the corresponding combined word appearing in the text to be processed.

[0076] Determine the heat value of each combined word respectively, and the heat value refers to the frequency of the combined word appearing in the text to be processed.

[0077] Reference Figure 3 , in one embodiment, the determining the heat value of each combined word in the at least one combined word includes:

[0078] S301, determine the value of the first parameter and the value of the second parameter of each combined word in the at least one combined word; the first parameter represents the number of times the corresponding combined word appears in the text to be processed; the second parameter represents the distribution of the corresponding combined word in the text to be processed;

[0079] The embodiment of the present invention uses the TF-IDF algorithm to calculate the heat value of the combination word. The TF-IDF algorithm represents the importance of the word. Here, the first parameter can be the term frequency (TF), and the second parameter can be the inverse document frequency index (IDF). The term frequency (TF) and the inverse document frequency (IDF) are associated with the importance of the word. The weight of a word is proportional to the frequency of the word appearing in the text, and inversely proportional to the frequency of the word appearing in all texts. The algorithm not only highlights high-frequency words, but also eliminates the influence of common words. It has a good effect and is the most widely used method for extracting text feature words.

[0080] The first parameter refers to the number of times a combination word appears in the text to be processed. The more times it appears, the greater the relevance between the combination word and the topic of the text to be processed.

[0081] Here, the calculation formula of the first parameter is as follows:

[0082]

[0083] In the formula, Count(word) represents the number of times a certain combination word appears in the text, and the denominator refers to the sum of the number of times all combination words appear.

[0084] The second parameter indicates the distribution of the combination in the text. The frequencies of some combination in different types of text are different. The more texts a combination appears in, the worse its text differentiation ability is. Therefore, IDF is used to distinguish texts of different categories. The IDF of a combination can be obtained by dividing the total number of texts containing the combination by the total number of texts and then taking the logarithm.

[0085] Here, the calculation formula of the second parameter is as follows:

[0086]

[0087] In the formula, Count(docs) is the total number of all texts, and the denominator is the total number of texts containing the word combination. The smaller Count(word, docs), the better the classification of the word combination.

[0088] S302: Determine the popularity value of the corresponding combination word based on the value of the first parameter and the value of the second parameter.

[0089] Multiply the value of the first parameter by the value of the second parameter to obtain the popularity value of the corresponding combination word.

[0090] Here, the calculation formula of the heat value is as follows:

[0091] Weight(word)=TF(word)*IDF(word)

[0092] In the formula, Weight is the heat value of the combination word, and Weight represents the weight of the combination word. The essence of the TF-IDF algorithm is to take TF as the weight of the combination word, and then use the IDF function to multiply TF to adjust the weight. If a combination word appears frequently in a text and few texts contain the combination word, then the combination word can better represent the characteristics of the text.

[0093] refer to Figure 4 In one embodiment, determining the popularity value of each combination word in the at least one combination word includes:

[0094] S401, clustering each of the at least one combination of words to obtain at least two combination of words sets; each of the at least two combination of words sets corresponds to a different risk level.

[0095] When there are a large number of combined words obtained from the text to be processed, if the heat values ​​are calculated for all the combined words, it will take a long time, and there is no need to perform heat analysis on some of the combined words.

[0096] In this case, each combination word in at least one combination word can be clustered. Clustering can group the same or similar objects together. In an embodiment of the present invention, clustering can obtain at least two combination word sets, and different combination word sets correspond to different risk levels.

[0097] For example, suppose there are three types of word combinations: intelligence, tip-off, and gossip. Among them, intelligence texts belong to high-risk public opinion and require business personnel to pay special attention. Tip-off texts belong to medium-risk public opinion, and gossip texts belong to low-risk public opinion. Perform cluster analysis on at least one word combination, and divide the words in at least one word combination into the word combination set of the corresponding category.

[0098] refer to Figure 5 , clustering each of the at least one combined word to obtain at least two combined word sets, including:

[0099] S501, vectorize each combined word in the at least one combined word to obtain at least one corresponding vector.

[0100] To cluster compound words, the compound words need to be vectorized first, and at least one compound word is clustered based on the distance between the vectors.

[0101] Here, the embodiment of the present invention can use the Doc2vec model to vectorize each of the at least one combination of words. Compared with long texts, short text public opinion analysis is more difficult. Due to the characteristics of short texts such as small number of words, high text generalization, high structural freedom, and rich information, it is more difficult to obtain key information from short texts when performing public opinion analysis than long texts. The embodiment of the present invention uses the Doc2vec model suitable for short texts to vectorize the combination of words, which has better results.

[0102] Doc2vec is proposed on the basis of the Word2vec model, so it also has the advantages of the Word2vec model. The advantage of Doc2vec is that it has good results when processing sentences of different lengths. Doc2vec is an unsupervised learning algorithm, with two core models, PV-DM (Distributed Memory version of Paragraph Vectors) and PV-DBOW (Distributed Bag of Words version of Paragraph Vector). It can train vectors with strong representation capabilities to represent words and sentences, and convert sentence vectors into dense real number vectors. The Doc2vec model structure overcomes the shortcomings of the bag of words model. The training method of the Doc2vec model is very similar to that of the Word2vec model. The core idea of ​​training word vectors is to predict the probability of the current word based on the context words. Similarly, the Doc2vec model can be trained in the same way.

[0103] Each combined word in at least one combined word is vectorized to obtain at least one vector, and one combined word corresponds to one vector.

[0104] S502, determining the distance value between each vector in the at least one vector and at least two set cluster centers; each cluster center in the at least two set cluster centers corresponds to a combination word set.

[0105] Each combination word set includes a corresponding cluster center, which is a pre-set vector. By calculating the distance between each vector in at least one vector and each cluster center, the combination word set corresponding to the cluster center closest to the vector is determined as the combination word set corresponding to the combination word.

[0106] S503: Determine a combination word set corresponding to each combination word in the at least one combination word based on the distance value.

[0107] Calculating the distance value is actually calculating the similarity between vectors, calculating the distance value between each vector in at least one vector and each cluster center, and determining the combination word set corresponding to the cluster center with the closest vector distance value as the combination word set corresponding to the combination word. For example, assuming there is a vector a and 3 cluster centers, where the distance between vector a and cluster center A is 1, the distance between vector a and cluster center B is 2, and the distance between vector a and cluster center C is 3, where the distance between vector a and cluster center A is the shortest, the combination word set corresponding to cluster center A is determined as the combination word set of the combination word corresponding to vector a.

[0108] Existing methods for extracting hot words from public opinion can be mainly divided into supervised learning methods and unsupervised learning methods. Supervised hot word extraction methods require a document set that has been segmented and annotated to train the classification model in advance. Supervised hot word extraction methods include decision trees based on information gain, naive Bayes models, and other methods. Supervised learning methods for extracting hot words from public opinion have high manual annotation costs and are difficult to achieve automatic and real-time public opinion monitoring. The model needs to be trained in advance and is not suitable for large-scale keyword extraction applications. The training time required is long, and the information gain of each feature needs to be calculated. When the sample categories are unbalanced, feature selection will be biased towards classes with large data volumes.

[0109] Unsupervised hot word extraction methods include: TF-IDF algorithm based on statistics, Topic Model model and TextRank based on graph model.

[0110] In practical applications, the K-means algorithm can be used to cluster each combination word in at least one combination word. The K-means algorithm uses the distance between samples as the clustering indicator to perform related clustering analysis activities. The K-means algorithm is a typical unsupervised clustering method that does not require manual labeling, which greatly reduces the cost of clustering.

[0111] The K-means algorithm requires pre-training of cluster centers. The training process includes the following steps:

[0112] Step 1: Select K samples in the data space as the initial centers, where each object represents a cluster center.

[0113] Step 2: For the data in the sample set, the belonging is determined by calculating the distance to the cluster center, and the sample is assigned to the class to which the closest cluster center belongs according to the closest distance criterion.

[0114] Step 3: Update the cluster center, take the mean corresponding to all objects in each category as the cluster center of the category, calculate the value of the objective function, and then continue to put in samples to calculate their distances, continue to classify according to the nearest distance, and recalculate the cluster center.

[0115] Step 4: Repeat step 2 to determine whether the cluster center and the set objective function value change after each iteration. When the cluster center and the set objective function value no longer change, the final cluster center is used as the set cluster center.

[0116] The present invention applies unsupervised learning algorithms for clustering, thus avoiding the high delay and cost caused by manual labeling and achieving the timeliness of extracting hot words in public opinion.

[0117] S402: Determine the heat value of the combination words in the combination word set with the highest risk in the at least two combination word sets.

[0118] The heat value is calculated for the combination words in the combination word set with the highest risk. For example, the intelligence category in the above embodiment belongs to high-risk public opinion and requires the focus of business personnel. Therefore, the combination words in the intelligence category are further analyzed for hot words.

[0119] S105, determining hot words in the text to be processed based on the heat value; the hot words represent combination words whose heat value is greater than a set value.

[0120] Here, the combination words whose heat value is greater than the set value can be determined as the hot words in the text to be processed, or the combination words with the top heat value can be determined as the hot words in the text to be processed according to the heat value.

[0121] refer to Figure 6 In one embodiment, the method further comprises:

[0122] S601, sorting the at least one combination word based on the popularity value to obtain a sorting result.

[0123] According to the heat value, at least one combination word is sorted in descending order.

[0124] S602: Display and output the sorting result.

[0125] Draw a hot word cloud chart based on the sorting results, and present the sorting results to business personnel in a visual way to help them adjust and control the marketing strategies corresponding to abnormal hot words.

[0126] The embodiment of the present invention performs word segmentation processing on the text to be processed to obtain at least two word segmentations, performs part-of-speech tagging on each of the at least two word segmentations, and combines the word segmentations in the at least two word segmentations that are marked with part-of-speech to obtain at least one combined word, wherein the combined word is composed of the word segmentations in the at least two word segmentations that are adjacent in position in the text to be processed. The heat value of each combined word in the at least one combined word is determined, and the hot words in the text to be processed are determined based on the heat value. The heat value represents the frequency of occurrence of the corresponding combined word in the text to be processed, and the hot word represents a combined word whose heat value is greater than a set value. The embodiment of the present invention prevents the key information from being disassembled during the word segmentation process by combining the word segmentations in the at least two word segmentations that are marked with part-of-speech, and the extracted hot words are more accurate, which is convenient for business personnel to adjust marketing strategies according to the hot words.

[0127] refer to Figure 7 , Figure 7 It is a schematic diagram of a hot word extraction process provided by an application embodiment of the present invention. The hot word extraction process includes: text acquisition, data preprocessing, feature engineering, text clustering, hot word extraction and public opinion warning.

[0128] First, the text to be processed is obtained through crawling technology. The text to be processed can come from public forums, netizens' comments, chat groups, etc.

[0129] The text to be processed is preprocessed, including corpus cleaning, word segmentation, part-of-speech tagging, stop word removal, part-of-speech secondary combination, and invalid word filtering. First, the corpus is cleaned to remove invalid information such as picture links, and then word segmentation and part-of-speech tagging are performed, and stop words are removed. Then the part-of-speech secondary combination is performed. This step is the uniqueness of the present invention. Since the key text feature in the marketing scenario is the combination of numerals + other parts of speech, such as "60 for auction", "199-120 coupons", "30 buy 300", etc., it is necessary to use the part-of-speech secondary combination to assist in identifying texts with special parts of speech. By combining the part-of-speech-marked part-of-speech part, it is possible to prevent the key information from being disassembled during the word segmentation process, and the extracted hot words are more accurate. Finally, invalid words are filtered.

[0130] Feature engineering mainly includes two steps: first, vectorizing the text, and then constructing word weights. Here, each of the at least one combined word can be vectorized through the Doc2vec model.

[0131] The processed texts are clustered and analyzed using the K-means algorithm to divide the texts into three categories: intelligence, tip-offs, and gossip. Among them, intelligence texts are high-risk public opinions and require business personnel to pay special attention to them. Tip-off texts are medium-risk public opinions, and gossip texts are low-risk public opinions. Therefore, we need to further analyze the hot words of high-risk intelligence texts.

[0132] Hot word extraction is divided into two steps. First, choose a suitable word heat measurement method to calculate the word heat. Here, TF-IDF technology can be used to calculate the word heat. After calculating the word heat, the words with higher heat are extracted as hot words according to the heat size.

[0133] Based on the hot word sorting results, a public opinion hot word cloud map is drawn, combined with the text clustering results to form a public opinion monitoring dashboard, and the public opinion hot word extraction results are visualized to help business personnel adjust and control the marketing strategies corresponding to abnormal hot words.

[0134] The application embodiment of the present invention is aimed at the marketing risk control scenario, and the secondary combination of parts of speech can prevent the key information from being disassembled during the word segmentation process, and the extracted hot words are more accurate, which is convenient for business personnel to adjust marketing strategies according to the hot words. In the public opinion scenario, Doc2Vec suitable for short texts is used to vectorize the text, which has better results. Unsupervised learning algorithms are applied for clustering to avoid the high cost of delays caused by manual labeling and realize the timeliness of public opinion hot word extraction. TF-IDF is used to score keywords, and a hot word cloud map is output based on the score.

[0135] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0136] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0137] It should be noted that the technical solutions described in the embodiments of the present invention can be arbitrarily combined without conflict.

[0138] In addition, in the embodiments of the present invention, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0139] refer to Figure 8 , Figure 8 is a schematic diagram of a hot word determination device provided by an embodiment of the present invention, such as Figure 8 As shown, the device includes: a word segmentation module, a tagging module, a combination module, a first determination module and a second determination module.

[0140] A word segmentation module is used to perform word segmentation on the text to be processed to obtain at least two word segments;

[0141] A tagging module, used for performing part-of-speech tagging on each of the at least two participles;

[0142] A combining module, used for combining the at least two participles with parts of speech marked thereon to obtain at least one combined word; the combined word is composed of the at least two participles that are adjacent in position in the text to be processed;

[0143] A first determination module is used to determine the popularity value of each combination word in the at least one combination word; the popularity value represents the frequency of occurrence of the corresponding combination word in the text to be processed;

[0144] The second determination module is used to determine the hot words in the text to be processed based on the heat value; the hot words represent the combination words whose heat value is greater than a set value.

[0145] In one embodiment, when the combining module combines the at least two participles with parts of speech marked to obtain at least one combined word, it is used to:

[0146] Combining the at least two participles with parts of speech marked based on a first setting rule to obtain at least one first combined word;

[0147] Based on the second setting rule, the first combination word in the at least one first combination word and the participle in the at least two participles are combined to obtain at least one second combination word; each second combination word in the at least one second combination word includes a first combination word.

[0148] In the above embodiment, each of the at least one combined word includes at least one numeral.

[0149] In one embodiment, when determining the popularity value of each combination word in the at least one combination word, the first determination module is used to:

[0150] Clustering each of the at least one combination of words to obtain at least two combination of words sets; each combination of words sets in the at least two combination of words sets corresponds to a different risk level;

[0151] Determine the heat value of the combination word in the combination word set with the highest risk in the at least two combination word sets.

[0152] In one embodiment, when clustering each of the at least one combined word to obtain at least two combined word sets, the first determining module is used to:

[0153] Vectorize each of the at least one combined word to obtain at least one vector;

[0154] Determine the distance value between each vector in the at least one vector and at least two set cluster centers; each cluster center in the at least two set cluster centers corresponds to a combination word set;

[0155] A combination word set corresponding to each combination word in the at least one combination word is determined based on the distance value.

[0156] In one embodiment, when determining the popularity value of each combined word in the at least one combined word, the first determining module is configured to:

[0157] Determine the value of a first parameter and a value of a second parameter of each of the at least one combination of words; the first parameter represents the number of times the corresponding combination of words appears in the text to be processed; the second parameter represents the distribution of the corresponding combination of words in the text to be processed;

[0158] The popularity value of the corresponding combination word is determined based on the value of the first parameter and the value of the second parameter.

[0159] In one embodiment, the device further comprises:

[0160] A sorting module, used for sorting the at least one combination word based on the popularity value to obtain a sorting result;

[0161] An output module is used to display and output the sorting results.

[0162] In actual application, the word segmentation module, labeling module, combination module, first determination module and second determination module can be implemented by a processor in an electronic device, such as a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU) or a programmable gate array (FPGA).

[0163] It should be noted that: the hot word determination device provided in the above embodiment only uses the division of the above modules as an example when determining hot words. In actual applications, the above processing can be assigned to different modules as needed, that is, the internal structure of the device is divided into different modules to complete all or part of the processing described above. In addition, the hot word determination device provided in the above embodiment and the hot word determination method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0164] Based on the hardware implementation of the above program modules and in order to implement the method of the embodiment of the present application, the embodiment of the present application also provides an electronic device. Fig. 9 Schematic diagram of the hardware structure of the electronic device of the present application embodiment. Fig. 9 As shown, the electronic equipment includes:

[0165] Communication interface, which can exchange information with other devices such as network equipment;

[0166] The processor is connected to the communication interface to realize information exchange with other devices, and is used to execute the method provided by one or more technical solutions of the electronic device side when running the computer program. The computer program is stored in the memory.

[0167] Of course, in actual applications, the various components in the electronic device are coupled together through the bus system. It is understandable that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig. 9 In the specification, various buses are labeled as bus systems.

[0168] The memory in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program used to operate on the electronic device.

[0169] It can be understood that the memory can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 130 described in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memories.

[0170] The method disclosed in the above embodiment of the present application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the program in the memory and completes the steps of the above method in combination with its hardware.

[0171] Optionally, when the processor executes the program, the corresponding processes implemented by the electronic device in each method of the embodiments of the present application are implemented, which will not be described in detail here for the sake of brevity.

[0172] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory storing a computer program, and the computer program can be executed by a processor of an electronic device to complete the steps of the aforementioned method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0174] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0175] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0176] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.

[0177] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0178] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0179] In addition, in the examples of the present application, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining hot words, characterized in that: The method comprises: Perform word segmentation on the text to be processed to obtain at least two word segments; Performing part-of-speech tagging on each of the at least two participles; Combining the at least two participles with parts of speech marked based on a first setting rule to obtain at least one first combination word; each combination word in the at least one first combination word includes at least one numeral; the first combination word is composed of the at least two participles that are adjacent in position in the to-be-processed text, and the first setting rule represents the parts of speech of the participles in the first combination word; Combining the first combination word in the at least one first combination word and the participle in the at least two participles based on the second setting rule to obtain at least one second combination word; each second combination word in the at least one second combination word includes a first combination word; Determine a heat value of each of the at least one combination of words; the heat value represents the frequency of occurrence of the corresponding combination of words in the text to be processed; Based on the heat value, hot words in the text to be processed are determined; the hot words represent combination words whose heat value is greater than a set value.

2. The method according to claim 1, characterized in that The determining the popularity value of each combined word in the at least one combined word includes: Clustering each of the at least one combination of words to obtain at least two combination of words sets; each combination of words sets in the at least two combination of words sets corresponds to a different risk level; Determine the heat value of the combination word in the combination word set with the highest risk in the at least two combination word sets.

3. The method according to claim 2, characterized in that The step of clustering each of the at least one combined word to obtain at least two combined word sets includes: Vectorize each of the at least one combined word to obtain at least one vector; Determine the distance value between each vector in the at least one vector and at least two set cluster centers; each cluster center in the at least two set cluster centers corresponds to a combination word set; A combination word set corresponding to each combination word in the at least one combination word is determined based on the distance value.

4. The method according to any one of claims 1 or 2, characterized in that: The determining the popularity value of each combined word in the at least one combined word includes: Determine the value of a first parameter and a value of a second parameter of each of the at least one combination of words; the first parameter represents the number of times the corresponding combination of words appears in the text to be processed; the second parameter represents the distribution of the corresponding combination of words in the text to be processed; The popularity value of the corresponding combination word is determined based on the value of the first parameter and the value of the second parameter.

5. The method according to claim 1, characterized in that The method further comprises: Sorting the at least one combination of words based on the popularity value to obtain a sorting result; The sorting result is displayed and outputted.

6. A hot word determination device, characterized in that: include: A word segmentation module is used to perform word segmentation on the text to be processed to obtain at least two word segments; A tagging module, used for performing part-of-speech tagging on each of the at least two participles; A combination module, configured to combine the at least two segmented words with parts of speech marked based on a first setting rule to obtain at least one first combination word; each of the at least one first combination word includes at least one numeral; the first combination word is composed of the segmented words in the at least two segmented words that are adjacent in position in the to-be-processed text, and the first setting rule represents the parts of speech of the segmented words in the first combination word; based on a second setting rule, combine the first combination word in the at least one first combination word and the segmented words in the at least two segmented words to obtain at least one second combination word; each of the at least one second combination word includes a first combination word; A first determining module, configured to determine a popularity value of each combined word in the at least one combined word; The heat value represents the frequency of occurrence of the corresponding combination word in the text to be processed; A second determination module, configured to determine a hot word in the text to be processed based on the heat value; The hot words represent combination words whose heat value is greater than a set value.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the hot word determination method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the hot word determination method according to any one of claims 1 to 5.

Images