Method and electronic device for extracting topic tags from text set

Abstract

The present application provides a method and electronic device for extracting topic tags from a text set, the method comprising: converting each text in the text set into a text vector; taking each text vector as the bottom-level cluster, performing bottom-up hierarchical clustering, and determining the topic tags of each layer of clusters; for any word, obtaining a cluster set corresponding to the word according to the cluster containing the vocabulary in the topic tag; the cluster set includes at least one cluster, each cluster containing at least one text; according to the cluster sets corresponding to different words and the keywords to be extracted, finding a target cluster set for keyword mapping; according to the topic tags corresponding to each cluster in the target cluster set, obtaining the topic tags related to the keywords extracted from the text set. The extraction of topic tags in this scheme is simpler.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method and electronic device for extracting topic tags from a text set. Background Art

[0002] The theme is the central idea of ​​the text, which summarizes and reflects the main body and core of the text content. The theme tag can succinctly summarize the main content of the text with a small number of words. In the era of information overload and rapid data growth, enterprises will accumulate massive text resources. When the amount of text is large and the source channels are wide, the data set contains content from different fields and types. In fact, we are faced with the problem of obtaining the theme of the text from multiple angles and understanding the relative relationship between these texts.

[0003] To extract topic tags from a large amount of text, we can directly use topic models, such as the common Latent Dirichlet Allocation (LDA), Probabilistic Latent Semantic Analysis (PLAS), etc. However, there are problems such as poor learning effect when there are short texts in the data set, and as a bag-of-words model, it cannot perceive advanced features such as word order, grammar, and semantics, and has limited scalability and cannot introduce newer neural network technologies.

[0004] Because the data set may contain text content in different directions, it is impossible to know whether different texts are describing similar or different content without prior knowledge. The first approach that should be taken is to use classification or clustering algorithms to classify the content. As a supervised learning, the classification algorithm needs to set labels in advance, that is, the extracted keywords, which is a method that puts the cart before the horse. Clustering can be used to pre-process the text and then vectorize it. The algorithm divides the text into multiple clusters and then further analyzes each cluster separately to further extract the keywords. However, the cutting theme of this method needs to be determined according to the result of the vector representation, which is difficult to operate in a targeted manner. Traditional K-means clustering algorithms also need to determine the number of clusters for clustering, which is difficult to determine when the number of splits in the data set is uncertain. Summary of the invention

[0005] The embodiment of the present application provides a method for extracting topic tags from a text set, thereby reducing the difficulty of extracting topic tags.

[0006] The present application also provides a method for extracting topic tags from a text set, including:

[0007] Convert each text in the text set into a text vector;

[0008] Take each text vector as the bottom-level cluster, perform bottom-up hierarchical clustering, and determine the topic label of each layer of clusters;

[0009] For any word, according to the cluster containing the word in the topic tag, a cluster set corresponding to the word is obtained; the cluster set includes at least one cluster, and each cluster includes at least one text;

[0010] According to the cluster sets corresponding to different words and the keywords to be extracted, find the target cluster set of the keyword mapping;

[0011] According to the topic label corresponding to each cluster in the target cluster set, a topic label related to the keyword extracted from the text set is obtained.

[0012] In one embodiment, converting each text in the text set into a text vector includes:

[0013] Perform word segmentation on each text in the text set;

[0014] Each text is converted into a text vector by using a trained encoder according to the words it contains.

[0015] In one embodiment, determining the topic labels of each layer of clusters includes:

[0016] For each cluster, the importance value of each word is calculated according to the word frequency value of each word in the cluster;

[0017] According to the importance value of each word in the cluster, a preset number of words with the largest importance values ​​are selected as topic labels corresponding to the cluster.

[0018] In one embodiment, the step of calculating the importance value of each word according to the frequency value of each word in the cluster includes:

[0019] For each word in the cluster, the importance value of the word is calculated according to the inverse document frequency value of the word in the text set and the word frequency value in the cluster.

[0020] In one embodiment, the importance value of the word is calculated based on the inverse document frequency value of the word in the text set and the word frequency value in the cluster, including:

[0021] Multiplying the inverse document frequency value of the word in the text set and the word frequency value in the cluster to obtain a word frequency-inverse document frequency value;

[0022] The word frequency-inverse document frequency value is used as the importance value of the word.

[0023] In one embodiment, the topic tag includes a plurality of topic words, and for any word, obtaining a cluster set corresponding to the word according to a cluster containing the word in the topic tag includes:

[0024] For each keyword, the clusters whose keyword labels contain the keyword are divided into the same cluster set;

[0025] If any two subclusters are merged into a new cluster and the subject labels of the two subclusters are the same, the two subclusters are deleted from the cluster set corresponding to the subject words according to the subject words contained in the subject labels.

[0026] In one embodiment, after obtaining the cluster set corresponding to the word, the method further includes:

[0027] The clusters in the cluster set are sorted in descending order according to the number of texts contained in each cluster in the cluster set.

[0028] In one embodiment, obtaining the topic label related to the keyword extracted from the text set according to the topic label corresponding to each cluster in the target cluster set includes:

[0029] According to the ranking order of each cluster in the target cluster set, the topic label of the cluster with the highest ranking is used as the topic label related to the keyword extracted from the text set.

[0030] In one embodiment, after finding the target cluster set of the keyword mapping, the method further includes:

[0031] According to the text contained in each cluster in the target cluster set, the text content related to the keyword extracted from the text set is obtained.

[0032] The present application also provides an electronic device, the electronic device comprising:

[0033] processor;

[0034] a memory for storing processor-executable instructions;

[0035] The processor is configured to execute the above method for extracting topic tags from a text set.

[0036] The technical solution provided by the above-mentioned embodiment of the present application converts each text in the text set into a text vector, adopts a hierarchical clustering method, and determines the topic label of each layer of clusters. For each word, a cluster set corresponding to each word is constructed according to the cluster containing the word in the topic label, and then the target cluster set of keyword mapping can be found. According to the topic label corresponding to each cluster in the target cluster set, the topic label related to the keyword is obtained. Because both long and short texts are represented as semantic vectors of fixed length, both short texts and long texts can be processed and mined in a clustering manner, and there are no problems that traditional topic models have when facing short texts. By adopting a bottom-up clustering method, there is no need to set the number of clusters for clustering, the calculation of topic labels is convenient, and the search of the target cluster set for keyword mapping is convenient. The topic labels of multi-level clusters (i.e., different granularities) can be obtained, and the calculation is simpler and the scope of application is wider. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0038] Figure 1 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0039] Figure 2 It is a flowchart of a method for extracting topic tags from a text set provided by an embodiment of the present application;

[0040] Figure 3 It is a schematic diagram of the principle of hierarchical clustering provided in an embodiment of the present application;

[0041] Figure 4 yes Figure 2 Detailed flowchart of step S220 in the corresponding embodiment;

[0042] Figure 5 It is a block diagram of a device for extracting topic tags from a text set provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0044] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0045] Figure 11 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 can be used to execute the method for extracting topic tags from a text set provided in an embodiment of the present application. Figure 1 As shown, the electronic device 100 includes: one or more processors 102, and one or more memories 104 storing processor executable instructions. The processor 102 is configured to execute the method for extracting topic tags from a text set provided in the following embodiment of the present application.

[0046] The processor 102 can be a gateway, a smart terminal, or a device including a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic device 100 and control other components in the electronic device 100 to perform desired functions.

[0047] The memory 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the method for extracting topic tags from a text set described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0048] In one embodiment, Figure 1 The electronic device 100 may also include an input device 106, an output device 108, and a data acquisition device 110, and these components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device 100 may also have other components and structures as required.

[0049] The input device 106 may be a device for a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc. The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, etc. The data acquisition device 110 may acquire an image of an object, and store the acquired image in the memory 104 for use by other components. Exemplarily, the data acquisition device 110 may be a camera.

[0050] In one embodiment, the various components in the example electronic device 100 for implementing the method of extracting topic tags from a text set according to an embodiment of the present application may be integrated or dispersed, such as integrating the processor 102, memory 104, input device 106, and output device 108 into one, while separating the data acquisition device 110.

[0051] In one embodiment, the example electronic device 100 for implementing the method of extracting topic tags from a text set according to an embodiment of the present application may be implemented as an intelligent terminal such as a smart phone, a tablet computer, a desktop computer, a server, or the like.

[0052] Figure 2 FIG. 1 is a flow chart of a method for extracting topic tags from a text set provided by an embodiment of the present application. Figure 2 As shown, the method includes the following steps S210-S250.

[0053] Step S210: Convert each text in the text set into a text vector.

[0054] Extracting topic tags from a text set, for example, can be to extract topic tags from the e-commerce reviews of a computer accessories store, so as to determine users' evaluations of different products and sales communications, and further improve their own service quality.

[0055] A text set refers to a collection of a large number of texts. For example, a text can be a review. Assume that the computer accessories store mentioned above sells CPUs, motherboards, graphics cards, memory and other accessories of different brands, which are divided into good and bad reviews. There are 100,000 reviews in total, and the text set can be these 100,000 reviews.

[0056] The text vector is used to represent the text features of each text, and a text can get a corresponding text vector. Specifically, you can choose an algorithm that can encode text into a fixed-length vector, such as Doc2Vec based on bag of words, or RNN (recurrent neural network) and Attention model based on deep learning, so as to convert the text into a text vector.

[0057] In one embodiment, each text in the text set may be firstly segmented, and then each text may be converted into a text vector through a trained encoder according to the words contained in each text.

[0058] Word segmentation can be performed using an existing word segmenter. The encoder can be Doc2Vec based on bag of words, or trained based on deep learning algorithms such as RNN (recurrent neural network) and Attention. The training dataset can be based on general or industry pre-trained word vectors. When the dataset has labels, use labels. If there are no labels, use contrastive learning or autoencoder algorithms. All texts in the dataset are numbered, and each text is represented as a vector using an encoder. This will give a set of all text vectors. Assuming there are N texts in the text set, there can be N text vectors.

[0059] Step S220: Take each text vector as the bottom-level cluster, perform bottom-up hierarchical clustering, and determine the topic label of each layer of cluster.

[0060] Hierarchical clustering can also be called hierarchical clustering, where each text vector is considered as an independent cluster at the bottom level, such as Figure 3 As shown in the figure, 1, 2, 3, 4, and 5 can represent the text vector of a text respectively, and each is an independent cluster. Calculate the distance between two clusters and find the two clusters with the smallest distance, such as 1 and 2, 4 and 5. Cluster 1 and cluster 2 are merged into a new cluster 6, and cluster 4 and cluster 5 are merged into a new cluster 7. In this way, three clusters 3, 6, and 7 are obtained. The merging can continue until the distance threshold condition is met or the set number of categories is reached.

[0061] In the hierarchical clustering process, the text numbers contained in each cluster can be continuously saved, and finally a tree-like clustering structure T is obtained, similar to Figure 3 As shown, the root node 9 contains all the text. At the beginning of the hierarchical clustering and each time the clusters are merged, the topic labels of each layer of clusters can be determined. Figure 3 For example, cluster 1, cluster 2, cluster 3, cluster 4 and cluster 5 represent the text vectors of text A, text B, text C, text D and text E respectively, so the topic label of cluster 1 is also the topic label of text 1, the topic label of cluster 2 is also the topic label of text 2, and so on. Cluster 6 is the merger of cluster 1 and cluster 2. Cluster 6 contains text 1 and text 2, so the topic label of cluster 6 is the common topic label of text 1 and text 2. Cluster 7 contains text 4 and text 5, so the topic label of cluster 7 is the common topic label of text 4 and text 5, and so on. The topic label is composed of K topic words, K is a hyperparameter, which can be adjusted according to the number of topic words you want to extract, generally 5-10. For example, the topic label can be [CPU, motherboard, memory, customer service, excellent].

[0062] It should be noted that in the clustering structure T, each node is a cluster of a text set, with K subject words attached. These subject words actually describe the core themes of several texts here, and can be used to describe the content expressed by the texts in this set. The cluster of the root node represents the entire text set, and the subject words therein are also representative subject words of all texts. Therefore, according to needs, in one embodiment, the subject label of the root node can be directly extracted as the subject label of the entire text set.

[0063] Step S230: for any word, according to the cluster containing the word in the topic tag, obtain a cluster set corresponding to the word; the cluster set includes at least one cluster, and each cluster contains at least one text.

[0064] For each word, a corresponding cluster set can be constructed. The word can appear in the topic label. For a word, if the topic label of a cluster contains the word, then the cluster will be added to the cluster set corresponding to the word. The cluster set can be considered as a collection of several clusters, and the topic labels of this cluster contain the same topic word.

[0065] For example, suppose a cluster C1 has topic labels [A, B, C, D, E], a cluster C2 has topic labels [A, C, D, E, F], a cluster C3 has topic labels [A, C, E, F, G], and a cluster C4 has topic labels [A, B, E, F, G]. For word A, there are clusters C1, C2, C3, and C4 whose topic labels include word A, so we can construct a cluster set (C1, C2, C3, and C4), for word B, we can construct a cluster set (C1, C4); for word C, we can construct a cluster set (C1, C2, C3); for word D, we can construct a cluster set (C1, C2), and so on. For each word, we can construct its corresponding cluster set.

[0066] Step S240: Finding a target cluster set for keyword mapping according to cluster sets corresponding to different words and keywords to be extracted.

[0067] Keywords can be keywords in the relevant information that the user wants to know, which can be input by the user or sent by other devices. When the store owner wants to know the evaluation of CPU, the keyword can be "CPU". The computer receives the keywords to be extracted and can find the cluster set corresponding to the keywords according to the cluster sets corresponding to different words. For the purpose of distinction, the cluster set corresponding to the keyword can be called the target cluster set.

[0068] Step S250: obtaining a topic label related to the keyword extracted from the text set according to the topic label corresponding to each cluster in the target cluster set.

[0069] Assume that the target cluster set C = [C_1, C_2, ..., C_n]. For any node cluster C_i, it includes K subject words. In addition to the query keyword, the remaining K-1 subject words can also be returned. The remaining K-1 subject words are supplementary descriptions of the keyword. Therefore, n groups of subject tags related to the keyword can be extracted in the end.

[0070] Since the text numbers of each cluster can be saved in the hierarchical clustering stage, the text content related to the keyword extracted from the text set can also be obtained according to the text contained in each cluster in the target cluster set. This text content can include the text contained in each cluster in the target cluster set.

[0071] Since the topic label of each cluster describes the core content of the cluster, when the target cluster set |C|=0, it means that there is no text in the text set that describes information related to the keyword. When the target cluster set C is not 0, each cluster in C can obtain a description of the same keyword at different granularities.

[0072] Generally speaking, the topic of clustering itself is determined by the generated sentence vector. The encoder algorithm generally ensures that similar texts will be closer, but the dimensions between different topics are difficult to operate. For example, a data set covers four types of content at the same time [Chinese reading, Chinese writing, English reading, English writing], assuming that the number of texts contained in the four categories is similar. When the data set is clustered into two categories, it is difficult to guarantee that the result is [Chinese reading + Chinese writing, English reading + English writing] or [Chinese reading + English reading, Chinese writing + English writing]. Retaining the results of each layer of hierarchical clustering can solve this problem to a certain extent. For example, if the clustering result is [Chinese reading + Chinese writing, English reading + English writing], it is hoped that "reading" will be used as a topic keyword for query, and two cluster sets [Chinese reading, English reading] can still be queried at a relatively deep position in the tree structure, and Chinese and English are used as auxiliary subject words to supplement the text in the cluster.

[0073] The technical solution provided by the above-mentioned embodiment of the present application converts each text in the text set into a text vector, adopts a hierarchical clustering method, and determines the topic label of each layer of clusters. For each word, a cluster set corresponding to each word is constructed according to the cluster containing the word in the topic label, and then the target cluster set of keyword mapping can be found. According to the topic label corresponding to each cluster in the target cluster set, the topic label related to the keyword is obtained. Because both long and short texts are represented as semantic vectors of fixed length, both short texts and long texts can be processed and mined in a clustering manner, and there are no problems that traditional topic models have when facing short texts. By adopting a bottom-up clustering method, there is no need to set the number of clusters for clustering, the calculation of topic labels is convenient, and the search of the target cluster set for keyword mapping is convenient. The topic labels of multi-level clusters (i.e., different granularities) can be obtained, and the calculation is simpler and the scope of application is wider.

[0074] In one embodiment, after constructing the cluster set corresponding to each word in step 230, the clusters in the cluster set may be sorted in descending order according to the number of texts contained in each cluster in the cluster set.

[0075] That is to say, for a cluster set, the cluster with more texts can be placed in front, and the cluster with less texts can be placed in the back. For example, assuming there are m words, we can get m cluster sets [C_1, C_2, ..., C_m], where C_i represents the cluster set corresponding to word i. For any cluster set C_i = [C_i1, C_i2, ..., C_il], sort in descending order according to the number of texts contained in C_ij. That is, cluster C_i1 has the most texts, and C_il has the least texts.

[0076] The clusters that are closer to the front in the target cluster set C contain more texts, so their subject labels have stronger generalized semantics, while the clusters that are closer to the back will explain the content contained in the cluster set from a more detailed perspective. Therefore, in one embodiment, the subject label of the cluster that is most closely ranked can be used as the subject label related to the keyword extracted from the text set according to the order of ranking of each cluster in the target cluster set. The cluster that is most closely ranked can be considered as the cluster that contains the largest number of texts in the target cluster set. The subject label of this cluster is more generalized and can be used as a subject label related to keywords. As needed, the subject label of the cluster that is closer to the back can also be selected, so as to obtain the subject label related to keywords from the corresponding granularity as needed.

[0077] In order to simplify the cluster set, the above step S230 specifically includes: for each keyword, the clusters whose topic tags contain the keyword are divided into the same cluster set; if any two sub-clusters are merged into a new cluster, and the topic tags of the two sub-clusters are the same, based on the keyword contained in this topic tag, the two sub-clusters can be deleted from the cluster set corresponding to the keyword.

[0078] Still Figure 3 For example, suppose clusters 1 and 2 are merged into cluster 6. Clusters 1 and 2 can be called subclusters, and cluster 6 can be called a new cluster. If the topic labels extracted from clusters 1 and 2 are exactly the same, it means that clusters 1 and 2 describe exactly the same content. Clusters 1 and 2 will be merged into a larger cluster 6, and the topic labels of cluster 6 will be exactly the same as those of clusters 1 and 2. Therefore, only cluster 6 needs to be retained, and clusters 1 and 2 can be deleted.

[0079] In one embodiment, if Figure 4 As shown, the above step S220 determines the topic label of each layer of clusters, which specifically includes steps S221 and S222.

[0080] Step S221: For each cluster, the importance value of each word in the cluster is calculated based on the word frequency value of each word in the cluster.

[0081] Among them, the word frequency value of a word refers to the frequency of the word appearing in the text contained in the cluster, that is, the number of times the word appears divided by the total number of words in the text. The importance value is used to characterize the importance of the word in the cluster. The larger the word frequency value, the greater the importance value. In one embodiment, the importance value can be directly represented by the word frequency value. In another embodiment, the importance value of the word can also be calculated for each word in the cluster based on the inverse file frequency value (IDF value) of the word in the text set and the word frequency value (TF value) in the cluster. Specifically, the inverse file frequency value of the word in the text set and the word frequency value in the cluster can be multiplied to obtain the word frequency-inverse file frequency value (TF-IDF value). The word frequency-inverse file frequency value of the word is used as the importance value of the word in the cluster.

[0082] Step S222: According to the importance value of each word in the cluster, a preset number of words with the largest importance values ​​are selected as topic labels corresponding to the cluster.

[0083] The preset number can be adjusted according to the number of subject words you want to extract, which is generally 5-10. The 5-10 words with the largest importance values ​​in a cluster can be used as the subject labels of the cluster. In one embodiment, the 5-10 words with the largest TF-IDF values ​​in the cluster can be used as the subject labels of the cluster. In the hierarchical clustering process, for each cluster, the K words with the largest TF-IDF can be selected as the subject labels of the cluster and saved in each node of the cluster structure T. Because a bottom-up clustering method is used, in the process of calculating the TF value, to calculate the number of times a word appears, you only need to add the number of times the word appears in the two subclusters, and the total number of words is the union of the subcluster set, so the calculation of the cluster's subject label is convenient.

[0084] The following is an example of a practical application scenario.

[0085] Extract topic tags from the e-commerce reviews of a computer accessories store. Analyze users' reviews of different products and sales communications to further improve the quality of their own services.

[0086] Assume that the store has 100,000 reviews, selling CPUs, motherboards, graphics cards, memory and other accessories of different brands, which are divided into good and bad reviews. Use Doc2Vec unsupervised training to construct the encoder, and set the number of extracted keywords K = 5. Through the above steps S210-S230, the tree-like clustering structure T and the cluster set corresponding to each word (i.e., mapping M) are obtained.

[0087] At this point, we can find that the root node in T extracts the keywords [CPU, motherboard, memory, customer service, excellent]. Through these most general keywords, we can analyze that all the reviews mainly include CPU, motherboard and memory reviews at the product level. There are also many reviews discussing customer service behaviors. Overall, we can feel that they are all good reviews. Of course, buyers may not directly enter the words "good review" and "bad review" in the review content. We can only feel and analyze from the side. For example, the word "excellent" in this example represents the buyer's overall feeling about the product.

[0088] When the store owner wants to know the CPU-related reviews, he finds the cluster set C (C = [C_1, C_2, ..., C_n]) corresponding to the CPU in the mapping M. Among them, C_1 is the relatively shallowest cluster, and most of the CPU-related reviews can be obtained through the text number contained in this node. It is found that the theme label corresponding to cluster C_1 is [CPU, motherboard, customer service, excellent, INTEL]. Analysis shows that most of the reviews with CPU also contain the word motherboard. After all, e-commerce often sells CPU + motherboard sets. The CPUs sold by the store are of course AMD and INTEL products. Here, we observe that the theme label of cluster C_2 is [CPU, motherboard, customer service, excellent, AMD]. Most of the AMD reviews are in it, but it is ranked behind because of its lower sales volume relative to INTEL.

[0089] For some keywords, such hierarchical relationships can be directly observed. However, it can be found that the products sold by e-commerce actually include graphics cards, but they do not appear in the shallowest topic tags. In fact, including the above C_1 and C_2 clusters, both contain some graphics card or other accessories reviews, but most of them are still related to the CPU, so it can be considered that the extracted descriptions are still about the CPU. We can further query the cluster set corresponding to "graphics card" and find that the topic tags of its first cluster are [graphics card, ASUS, available, difficult to buy, power supply]. It can be analyzed from the side that ASUS graphics cards have a high sales volume, but there are fewer graphics cards in the market now.

[0090] It is found that the main information is now in the positive reviews. When the store wants to know the negative reviews to improve its own products and services, it can use some common negative review words as keywords for search based on experience. For example, directly search for "damaged". If you find that the theme label of the first cluster in the cluster set is [chassis, damaged, negative review, packaging, problem], although you don't know the number, you can know that the packaging of some chassis is damaged. According to the text number contained in this cluster, you can directly locate these evaluation texts. At the same time, you can check the remaining clusters to find out other damaged products and corresponding reviews.

[0091] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of extracting topic tags from a text set of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of extracting topic tags from a text set of the present application.

[0092] Figure 5 An apparatus for extracting topic tags from a text set is shown in an embodiment of the present application. The apparatus includes: a vector conversion module 510 , a hierarchical clustering module 520 , a mapping determination module 530 , a cluster set determination module 540 and a tag extraction module 550 .

[0093] A vector conversion module 510, for converting each text in the text set into a text vector;

[0094] A hierarchical clustering module 520, for taking each text vector as the bottom-level cluster, performing bottom-up hierarchical clustering, and determining a topic label for each level of cluster;

[0095] A mapping determination module 530 is used to obtain, for any word, a cluster set corresponding to the word according to the cluster containing the word in the subject tag; the cluster set includes at least one cluster, and each cluster includes at least one text;

[0096] A cluster determination module 540 is used to find a target cluster for keyword mapping according to clusters corresponding to different words and keywords to be extracted;

[0097] The tag extraction module 550 is used to obtain the topic tags related to the keyword extracted from the text set according to the topic tags corresponding to each cluster in the target cluster set.

[0098] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method for extracting topic tags from a text set, and will not be repeated here.

[0099] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0101] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

Claims

1. A method for extracting topic tags from a text set, characterized in that: include: Convert each text in the text set into a text vector; Take each text vector as the bottom-level cluster, perform bottom-up hierarchical clustering, and determine the topic label of each layer of clusters; For any word, according to the clusters containing the word in the subject tag, a cluster set corresponding to the word is obtained, including: for each subject word, the clusters containing the subject word in the subject tag are divided into the same cluster set; If any two subclusters are merged into a new cluster and the subject labels of the two subclusters are the same, the two subclusters are deleted from the cluster set corresponding to the subject words according to the subject words contained in the subject labels; the cluster set includes at least one cluster, and each cluster includes at least one text; According to the cluster sets corresponding to different words and the keywords to be extracted, find the target cluster set of the keyword mapping; According to the topic label corresponding to each cluster in the target cluster set, a topic label related to the keyword extracted from the text set is obtained.

2. The method according to claim 1, characterized in that The step of converting each text in the text set into a text vector includes: Perform word segmentation on each text in the text set; Each text is converted into a text vector by using a trained encoder according to the words it contains.

3. The method according to claim 1, characterized in that Determining the topic labels of each layer of clusters includes: For each cluster, the importance value of each word is calculated according to the word frequency value of each word in the cluster; According to the importance value of each word in the cluster, a preset number of words with the largest importance values ​​are selected as topic labels corresponding to the cluster.

4. The method according to claim 3, characterized in that The step of calculating the importance value of each word according to the word frequency value of each word in the cluster includes: For each word in the cluster, the importance value of the word is calculated according to the inverse document frequency value of the word in the text set and the word frequency value in the cluster.

5. The method according to claim 4, characterized in that The step of calculating the importance value of the word according to the inverse document frequency value of the word in the text set and the word frequency value in the cluster includes: Multiplying the inverse document frequency value of the word in the text set and the word frequency value in the cluster to obtain a word frequency-inverse document frequency value; The word frequency-inverse document frequency value is used as the importance value of the word.

6. The method according to claim 1, characterized in that After obtaining the cluster set corresponding to the word, the method further includes: The clusters in the cluster set are sorted in descending order according to the number of texts contained in each cluster in the cluster set.

7. The method according to claim 6, characterized in that The step of obtaining the topic tags related to the keyword extracted from the text set according to the topic tags corresponding to each cluster in the target cluster set includes: According to the ranking order of each cluster in the target cluster set, the topic label of the cluster with the highest ranking is used as the topic label related to the keyword extracted from the text set.

8. The method according to claim 1, characterized in that After finding the target cluster set of the keyword mapping, the method further includes: According to the text contained in each cluster in the target cluster set, the text content related to the keyword extracted from the text set is obtained.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method for extracting topic tags from a text set as described in any one of claims 1-8.

Images