Text clustering method, text clustering device and text clustering system

Abstract

This application provides a text clustering method, a text clustering device, and a text clustering system. In this solution, the Word2vec model is combined with the TF-IDF algorithm to represent word vectors, enhancing the distinction between different texts. This approach not only takes advantage of word vectors but also incorporates the influence of words on text. The word vectors represented by the combination of the two serve as the input of the WMD algorithm, which is used as a similarity measurement algorithm in text clustering to improve the accuracy of text clustering.

Description

Technical Field

[0001] The present application relates to the technical field of text clustering, and in particular to a text clustering method, a text clustering device, a computer-readable storage medium, and a text clustering system. Background Art

[0002] With the increasing speed of information dissemination and the continuous expansion of cyberspace, the amount of information on the Internet is growing exponentially, and the amount of unstructured text data is also increasing. To effectively utilize the valuable information contained in text data, clustering algorithms are used to construct text clusters based on common features within text data for subsequent information processing and analysis, which is a key step in extracting valuable information. Using common distance algorithms to calculate text similarity in the clustering process results in low clustering accuracy. Summary of the Invention

[0003] The main purpose of this application is to provide a text clustering method, a text clustering device, a computer-readable storage medium, and a text clustering system, so as to at least solve the problem of low accuracy of text clustering in the prior art.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a text clustering method is provided, comprising: obtaining multiple original texts, preprocessing each of the original texts to obtain multiple target texts, wherein the preprocessing includes at least one of the following: word segmentation processing, stop word removal processing, and the target text includes multiple words; using the Word2vec model to convert the words in each of the target texts into word vectors, and using the TF-IDF algorithm to determine the weight value of the word, wherein the weight value is the importance of the word in the target text; using the word vector and the weight value as input of the WMD algorithm, determining the similarity between any two of the target texts, and clustering the target texts according to the similarity.

[0005] Optionally, a Word2vec model is used to convert the words in each target text into word vectors, and a TF-IDF algorithm is used to determine the weight values ​​of the words, including: constructing a Word2vec model, wherein the Word2vec model is trained using multiple groups of training data, each group of training data in the multiple groups of training data includes: historical words and historical word vectors corresponding to the historical words, and different groups of training data are obtained in different historical time periods; inputting the words into the Word2vec model, and obtaining the output of the Word2vec model as the word vector corresponding to the words; obtaining the word frequency and inverse document frequency of the words in the target text, wherein the inverse document frequency is used to characterize the importance of the words; and using the TF-IDF algorithm to determine the weight values ​​of the words based on the word frequency and the inverse document frequency, wherein the smaller the product of the word frequency and the inverse document frequency, the smaller the weight value of the word and the lower the importance, and the larger the product of the word frequency and the inverse document frequency, the larger the weight value of the word and the higher the importance.

[0006] Optionally, the TF-IDF algorithm is used to determine the weight value of the word, further comprising: according to the target formula:

[0007]

[0008] Determine the weight value of the word, where f represents the word, m represents the target text, W(f,m) represents the weight value of the word f in the target text m, TF(f,m) represents the number of times the word f appears in the target text m, N represents the total number of the target text, and n i represents the number of the target text m containing the word f.

[0009] Optionally, the word vector and the weight value are used as inputs of the WMD algorithm to determine the similarity between any two target texts, including: performing TF-IDF&Word2vec vectorization on the first target text according to multiple first word vectors in the first target text and the first TF-IDF value corresponding to each of the first word vectors to obtain a first target set, wherein the first target set includes multiple first target word vectors and weight values, and the first word vector and the first TF-IDF value are used as inputs of the WMD algorithm; performing T-IDF&Word2vec vectorization on the second target text according to multiple second word vectors in the second target text and the second TF-IDF value corresponding to each of the second word vectors F-IDF&Word2vec vectorization representation is used to obtain a second target set, wherein the second target set includes multiple second target word vectors and weight values, and the second word vectors and the second TF-IDF value are used as inputs of the WMD algorithm; in the process of calculating the similarity between the first target text and the second target text using the WMD algorithm, the cosine distance between the first target word vector and the second target word vector is calculated as the transfer cost, the TF-IDF value is used for weight allocation, and the minimum value of the sum of the products of the transfer cost and the allocated weight is used as the distance between the first target text and the second target text, and the distance is the similarity of the distance between the first target text and the second target text.

[0010] Optionally, in the process of calculating the similarity between the first target text and the second target text using the WMD algorithm, the cosine distance between the first target word vector and the second target word vector is calculated as the transfer cost, the TF-IDF value is used for weight assignment, and before the minimum value of the sum of the products of the transfer cost and the assigned weight is used as the distance between the first target text and the second target text, the method further includes: obtaining a corpus, wherein the corpus includes multiple dictionary words; selecting one of the dictionary words from the corpus in turn as the central dictionary word; respectively calculating the cosine distance between the non-central dictionary words in the corpus and the central dictionary word, wherein the cosine distance is the cosine value of the angle between the word vector corresponding to the non-central dictionary word in the corpus and the word vector corresponding to the central dictionary word; the non-central dictionary words whose cosine distance is within the target range are stored in the unrelated set of the central dictionary words, and the non-central dictionary words whose cosine distance is not within the target range are stored in the related set of the central dictionary words.

[0011] Optionally, in the process of calculating the similarity between the first target text and the second target text using the WMD algorithm, the cosine distance between the first target word vector and the second target word vector is calculated as the transfer cost, and before the TF-IDF value is used for weight allocation, the method also includes: in the case where some of the second target word vectors in the second target set are in the related set of the first target word vector, and some of the second target word vectors are in the irrelevant set of the first target word vector, calculating the cosine distance of the second target word vector in the related set of the first target word vector as the transfer cost for the weight allocation of the second target word vector, and using the TF-IDF value for weight allocation; calculating the second average value of the distance from the second target word vector in the irrelevant set of the first target word vector to the first target word vector as the transfer cost for the weight allocation of the second target word vector, and using the TF-IDF value for weight allocation; calculating the sum of the first data and the second data to obtain the WMD distance between the first target text and the second target text, the first data being the minimum value of the sum of multiple products of the cosine distances and the weight of the second target word vector, and the second data being the product of the second average value and the weight of the second target word vector.

[0012] Optionally, clustering the target texts according to the similarity includes: when using one target text for the first clustering, taking one target text as the first text cluster; when using the Nth target text for the Nth clustering, comparing the similarity of the Nth target text with each target text in the formed text clusters, where N≥2; when the similarity is greater than or equal to a similarity threshold, classifying the Nth target text into the text cluster; when the similarity is less than the similarity threshold, creating a new text cluster based on the Nth target text.

[0013] According to another aspect of the present application, a text clustering device is provided, comprising: a first acquisition unit, configured to acquire a plurality of original texts, pre-process each of the original texts, and obtain a plurality of target texts, wherein the pre-processing comprises at least one of the following: word segmentation processing and stop word removal processing, and the target texts comprise a plurality of words; a first processing unit, configured to convert the words in each of the target texts into word vectors using a Word2vec model, and to determine weight values ​​of the words using a TF-IDF algorithm, wherein the weight values ​​represent the importance of the words in the target texts; and a second processing unit, configured to use the word vectors and the weight values ​​as inputs to a WMD algorithm, determine the similarity between any two of the target texts, and cluster the target texts according to the similarity.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute any one of the text clustering methods.

[0015] According to another aspect of the present application, a text clustering system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the text clustering methods.

[0016] By applying the technical solution of this application, word vectors are represented by combining the Word2vec model with the TF-IDF algorithm to enhance the distinction between different texts. The advantages of word vectors are utilized while the influence of words on texts is also added. The word vectors represented by the combination of the two are used as the input of the WMD algorithm. The WMD algorithm is used as the similarity measurement algorithm in text clustering to improve the accuracy of text clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a mobile terminal for executing a text clustering method provided in an embodiment of the present application is shown;

[0019] Figure 2 A schematic diagram of a process of a text clustering method provided according to an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of a process for preprocessing original text is shown;

[0021] Figure 4 A spatial diagram showing Euclidean distance and cosine distance;

[0022] Figure 5 A schematic diagram showing a model for WMD to calculate document distances;

[0023] Figure 6 A schematic diagram showing the range of word embedding distances in the corpus;

[0024] Figure 7 A schematic diagram showing the distribution of distances between central dictionary words and non-central dictionary words in a corpus;

[0025] Figure 8 A schematic diagram of the process of clustering target texts based on similarity is shown;

[0026] Figure 9 The figure shows a structural block diagram of a text clustering device provided according to an embodiment of the present application.

[0027] The above drawings include the following reference numerals:

[0028] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0029] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] The main reasons for the low accuracy of text clustering are:

[0033] 1) The commonly used Word2vec text representation model can convert each feature word into a real vector of fixed dimension through training. However, a document is a collection of several words, and obtaining a word vector is only an intermediate process; the ultimate goal is to use the word vector to represent the document. Currently, common methods include averaging all word vectors in a document and clustering word vectors. Because different words contribute differently to a document, these methods ignore the impact of individual words on the document, resulting in inaccurate text representation and affecting subsequent clustering results.

[0034] 2) The WMD (Word Mover's Distance) similarity algorithm is used directly to measure text similarity, resulting in low calculation accuracy. This algorithm only uses the frequency information of characteristic words in the text to define the transfer cost between words. This makes it difficult to suppress the influence of high-frequency words on the text and does not consider keyword factors, resulting in low text clustering accuracy.

[0035] 3) When the WMD algorithm calculates text similarity, each word in a text sentence must be compared with all the words in the text to calculate the transfer cost, and then the weight is allocated according to the transfer cost. The calculation complexity is high, resulting in low calculation efficiency.

[0036] As introduced in the background technology, the accuracy of text clustering in the prior art is low. To solve the above problem, the embodiments of the present application provide a text clustering method, a text clustering device, a computer-readable storage medium and a text clustering system.

[0037] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0038] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a mobile terminal for a text clustering method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1Different configurations shown.

[0039] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the device information display method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] In this embodiment, a text clustering method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Figure 2 FIG. 1 is a flow chart of a text clustering method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0042] Step S201: obtaining a plurality of original texts, and preprocessing each of the original texts to obtain a plurality of target texts, wherein the preprocessing includes at least one of the following: word segmentation processing and stop word removal processing, and the target texts include a plurality of words;

[0043] Specifically, text corpora cannot be used as initial data for vectorized modeling; standardized data formats ensure smooth execution of subsequent steps. Therefore, further processing of text information is possible, including word segmentation and stop word removal.

[0044] Word segmentation is a method of reorganizing continuous words into a sequence of words according to specific rules. According to the structure of English sentences, the spaces between words separate the words in the sentence, and there is no need to perform word segmentation on the sentence specifically. In Chinese, there is no separator between words, so word segmentation technology needs to be used to preprocess Chinese. Chinese word segmentation is a basic task in natural language processing. Chinese word segmentation requires splitting each sentence into words in a sequence form. Due to the complexity of Chinese in terms of semantics and grammar, it is difficult for computers to effectively understand Chinese. Therefore, jieba word segmentation can be used to split the text. Jieba is a third-party Chinese word segmentation library for Python, mainly supporting three word segmentation modes: accurate mode, full mode, and search engine mode. The principle is to compare the segmented content with the word segmentation library using the Chinese word segmentation library, and use the graph structure and dynamic programming to find the segmentation combination with the highest probability. Any one of these word segmentation modes can be used in this solution.

[0045] After Chinese word segmentation, the document (text) will become a set containing multiple words. Word elements with weak semantic information and no real meaning in the set, or some special symbols that have no impact on the semantics of the document, etc., are called stop words. For example, modal particles such as "呢", "呦", "吧", etc., structural particles such as "的", "地", "了", etc., conjunctions such as "只是", "并且", "和", etc. Although these stop words appear more frequently in sentences, they have almost no information content. The existence of stop words occupies the system storage space, increases the index quantity, and thus reduces the processing speed. Therefore, these words should be deleted in the document, which not only speeds up the processing of the document but also makes subsequent analysis more convenient. Usually, the words or characters to be deleted are placed in the stop word list. Check whether the result after Chinese word segmentation is in the stop word list. If it exists, delete it. After word segmentation, read the stop word list stored locally and save the processed text information. The specific process of preprocessing the original text is as follows Figure 3 shown. First, start to obtain the original text, perform noise reduction processing on the original text (delete some predetermined words, preliminary data cleaning), then perform word segmentation processing to obtain multiple words. Read a word in sequence and determine whether the word is in the stop word list. If the word is not in the stop word list, read another word. If the word is in the stop word list, delete the word until all the words in the text have been preprocessed, and the preprocessing process ends.

[0046] Step S202, use the Word2vec model to convert the above-mentioned words in each of the above-mentioned target texts into word vectors, and use the TF-IDF algorithm to determine the weight value of the above-mentioned words, where the above-mentioned weight value is the importance degree of the above-mentioned words in the above-mentioned target text;

[0047] Specifically, the Word2vec model leverages contextual information and, through training, can transform each word into a fixed-dimensional real-number vector (i.e., a word vector). Similar words are also close in distance in the vector space, addressing the high-dimensional and sparse nature of the word vector matrix and its neglect of semantics, grammar, and contextual relationships. The TF-IDF algorithm calculates the weights of words in a text and enhances the distinction between different texts. Therefore, the TF-IDF algorithm is used to weight the word vectors output by the Word2vec model to address the issue of inaccurate text representation models.

[0048] In step S203 , the word vectors and the weight values ​​are used as inputs of the WMD algorithm to determine the similarity between any two target texts, and cluster the target texts according to the similarity.

[0049] Specifically, to improve the efficiency of text similarity calculation and enhance clustering accuracy, this solution can use the WMD algorithm to calculate the similarity between texts. Of course, to further improve the efficiency of text similarity calculation and enhance clustering accuracy, the WMD algorithm can also be improved and used to calculate the similarity between texts. The clustering algorithm can be any available clustering algorithm, such as the Single-Pass incremental clustering algorithm.

[0050] Through this embodiment, the word vector is represented by combining the Word2vec model with the TF-IDF algorithm to enhance the distinction between different texts. The advantages of word vectors are utilized while the influence of words on texts is also taken into account. The word vector represented by the combination of the two is used as the input of the WMD algorithm. The WMD algorithm is used as the similarity measurement algorithm in text clustering to improve the accuracy of text clustering.

[0051] After the original text has been preprocessed, the data set of the original text will represent multiple words, namely the target text. In order to effectively apply the set of target texts with this structure to subsequent clustering analysis, the text needs to be vectorized. In the specific implementation process, the Word2vec model is used to convert the above words in each of the above target texts into word vectors, and the TF-IDF algorithm is used to determine the weight values ​​of the above words. This can be achieved through the following steps: Construct a Word2vec model, wherein the above Word2vec model is trained using multiple sets of training data, and each set of training data in the above multiple sets of training data includes: historical words and the historical word vectors corresponding to the above historical words. The training data of different groups are obtained in different historical time periods; the words are input into the Word2vec model, and the output of the Word2vec model is the word vector corresponding to the words; the word frequency and inverse document frequency of the words in the target text are obtained, and the inverse document frequency is used to characterize the importance of the words; the TF-IDF algorithm is used to determine the weight value of the words according to the word frequency and the inverse document frequency, wherein the smaller the product of the word frequency and the inverse document frequency is, the smaller the weight value of the words is, and the lower the importance is; the larger the product of the word frequency and the inverse document frequency is, the larger the weight value of the words is, and the higher the importance is.

[0052] In this solution, the Word2vec model is used to convert the words in the target text into word vectors of the same dimension. The weights of the words in the target text can also be calculated, so that the target text can be represented using word vectors and weight values. The TF-IDF algorithm is used to weight the word vectors obtained by the Word2vec model. TF-IDF is then used as the weight for weight allocation in WMD. That is, the TF-IDF & Word2Vec text representation model is used as the input to the WMD distance similarity algorithm. This objectively reflects the differences between words, avoids the excessive influence of high-frequency words on the document itself, and further improves the accuracy of text clustering.

[0053] Inverse Document Frequency (IDF) is a measure of the general importance of a word. Its size is inversely proportional to the commonness of a word. It is calculated by dividing the total number of documents in the corpus by the number of documents in the corpus that contain the word, and then taking the logarithm of the quotient.

[0054] The gensim library provided by Python is a powerful NLP tool that includes the Word2vec model and the TF-IDF model. It can be directly called in the program according to user needs to train and load the model. The initial formula of the TF-IDF algorithm can be:

[0055]

[0056] Where f represents a word, m represents a target text, W(f,m) represents the weight of word f in target text m, TF(f,m) represents the number of times word f appears in target text m, N represents the total number of target texts, and n i Represents the number of target texts m containing word f.

[0057] In order to further ensure that the weight values ​​of the words obtained by the TF-IDF algorithm are more accurate, the TF-IDF algorithm is used to determine the weight values ​​of the above words. This can be achieved by the following steps: According to the target formula:

[0058]

[0059] Determine the weight value of the above word, where f represents the above word, m represents the above target text, W(f,m) represents the weight value of the above word f in the above target text m, TF(f,m) represents the number of times the above word f appears in the above target text m, N represents the total number of the above target text, and n i represents the number of target texts m containing the word f.

[0060] In this solution, the initial formula of the TF-IDF algorithm can be further normalized to obtain a target formula, so as to further use the target formula to ensure that the weight values ​​of the obtained words are more accurate.

[0061] The above TF-IDF&Word2Vec model represents that the input of the text is the original text M={m1,m2,m3,...,m n}, the output is the target text (text vector set), the specific steps are as follows:

[0062] Step 1: Input the original text and preprocess it. Each text becomes a set of words, i.e. m i ={f i1 ,f i2 ,f i3 ,...,f ij};

[0063] Step 2: The processed text set is trained with the Word2vec model. For each feature word (word) f ij You will get an N-dimensional word vector V(f ij );

[0064] Step 3: Use the TF-IDF formula to get the weight W(f ij );

[0065] Step 4: Based on the calculation results in Step 2 and Step 3, each text is vectorized and represented as m i ={(V(f i1 ),W(f i1 )),(V(f i2 ),W(f i2 )),...}.

[0066] Text clustering is a process of grouping disorganized raw data into distinct clusters. Text within the same cluster is close in distance, indicating high similarity, while text similarity between different clusters is low. Therefore, the quality of a similarity measurement algorithm plays a crucial role in determining the quality of clustering results. Common similarity measurement algorithms include cosine distance and Euclidean distance. The following describes these two methods and illustrates the advantages of the WMD algorithm used in this solution.

[0067] Suppose there are two words W1 and W2. After training with the Word2vec model, we get two p-dimensional word vectors V(W1) = (x1, x2, x3...x p ) T and V(W2)=(y1,y2,y3...y p ) T , two common ways to measure the distance between word vectors are:

[0068] (1) Cosine distance uses the cosine value of the angle between two vectors to measure the difference between them. The first formula for calculation is:

[0069]

[0070] (2) Euclidean distance: also known as Euclidean distance, it uses the absolute distance between two points in the coordinate system to measure the difference between two vectors. The specific calculation formula is:

[0071]

[0072] To better explain the difference between the two similarity metrics, the effects of the Euclidean distance and cosine distance between point P and point Q in three-dimensional space are as follows: Figure 4 shown.

[0073] A document (also called text) is a collection of several words. Obtaining a word vector is only an intermediate process, and using a word vector to represent a document is the ultimate goal. Currently, commonly used methods include averaging all word vectors in a document and clustering word vectors. The word vector mean is used to replace the entire document, and then the distance between documents is calculated based on the distance calculated by the word vector. Although this method takes into account the semantic characteristics of the document to a certain extent, different words appearing in a document have different contributions to the document. Therefore, the method of averaging word vectors instead of document vectors ignores the impact of individual words on the document itself. Therefore, whether using Euclidean distance or cosine distance to measure text similarity, the impact of individual words on the document is ignored, resulting in inaccurate calculation results and the subsequent clustering effect will also be affected. Therefore, this solution uses the word shift distance WMD text similarity measurement algorithm, which takes into account the contribution of all words to the text, improves the accuracy of the calculation, and thus improves the accuracy of text clustering.

[0074] The word shift distance WMD takes into account the contribution of all words in a document to the document in the similarity calculation, which can be simply understood as calculating the minimum distance for all words in one document to be transferred to all words in another document. The theoretical basis of this algorithm is the earth mover distance EMD, also known as the bulldozer distance. EMD is widely used in speech signal processing and image processing. Kusner et al. applied it to the field of NLP and proved through experiments that the algorithm is quite effective. The word vectors trained in the Word2vec model have an addition and subtraction relationship. Based on this relationship between word vectors, when calculating the distance between two documents P1 and P2, the basic idea of ​​WMD is to calculate the distance between any word w in document P1. i , which can be found in document P2 i The word w′ with the smallest distance j , that is, w i Move to w′ j The distance between the two documents is the smallest, or the cost of the transfer is the smallest. Then the distance between the two documents is the total distance of all the words in P1 transferred to all the words in P2, or the total cost. This method of calculating document distance is similar to EMD, and does not require hyperparameters and can be calculated directly. Suppose the WMD distance between two documents P1 and P2 is calculated, the model is as follows Figure 5 As shown, the specific algorithm process is as follows:

[0075] Assume that the word vector trained by Word2vec is an m-dimensional corpus and there are n words in the corpus dictionary, then the word vector matrix X∈R m×n The number of times word w(i) in document P appears in the document is t w(i) , then the third formula for the weight of word w(i) in P is:

[0076]

[0077] According to the third formula, it can be determined that the weight of the feature word (also called word) in the WMD algorithm is calculated using the word frequency.

[0078] Assume that there are two documents P1 and P2 represented by the Word2vec model. Use the Euclidean distance to calculate the distance between the i-th word in P1 and the j-th word in P2, which is the transfer cost mentioned in the algorithm. The specific calculation formula is: c(i,j) = ||x i -x j ||2.

[0079] Define a transfer matrix Τ∈R n×n , T ij represents the weight distribution of word i in P1 transferred to word j in P2. The smaller the distance between word i and word j, the more similar the two words are, and the greater the weight transferred from word i to word j. The total transfer cost from P1 to P2 can be expressed as the fifth formula:

[0080]

[0081] The WMD of the document is obtained by calculating the minimum sum of the transfer costs of the two document words, that is, converting the distance calculation into a linear programming problem and finding the minimum value of the fifth formula. This algorithm stipulates two constraints to avoid the extreme situation where one word in P1 corresponds to all the words in P2. i represents the weight of word i in P1, D′ j represents the weight of word j in P2. The sixth formula for calculating WMD is as follows:

[0082]

[0083] Assume that the word vector trained by Word2vec is d-dimensional. After the word w in document D and the word w' in D' are represented by the Word2vec model, the word vector is represented as V(w i ),V(w' j )∈R d The WMD distance between documents D and D' is transformed according to the sixth formula to obtain the seventh formula:

[0084]

[0085] Where c(i,j)=||V(w i )-V(w' j )||2, however, the seventh formula still has the following shortcomings:

[0086] (1) Each word in document D must calculate the transfer cost with all words in D', and then the weight is assigned according to the transfer cost, which has high computational complexity.

[0087] (2) WMD uses word frequency to measure the importance of words when constraining the transfer amount. It is difficult to suppress the impact of high-frequency words on documents and does not consider keyword factors.

[0088] In order to further improve the efficiency and accuracy of the WMD algorithm, the words can be divided into related sets and unrelated sets according to their distribution in the embedding space. In some embodiments, the above-mentioned word vectors and the above-mentioned weight values ​​are used as inputs of the WMD algorithm to determine the similarity between any two of the above-mentioned target texts, including: according to the multiple first word vectors in the first target text and the first TF-IDF values ​​corresponding to the above-mentioned first word vectors, the above-mentioned first target text is vectorized by TF-IDF&Word2vec to obtain a first target set, wherein the above-mentioned first target set includes multiple first target word vectors and weight values, and the above-mentioned first word vectors and the above-mentioned first TF-IDF value are used as the inputs of the above-mentioned WMD algorithm; according to the multiple second word vectors in the second target text and the first TF-IDF values ​​corresponding to the above-mentioned second word vectors, the above-mentioned first target text is vectorized by TF-IDF&Word2vec to obtain a first target set. The second TF-IDF value corresponding to the two word vectors is used to represent the second target text by TF-IDF&Word2vec vectorization to obtain a second target set, wherein the second target set includes multiple second target word vectors and weight values, and the second word vector and the second TF-IDF value are used as inputs of the WMD algorithm; in the process of calculating the similarity between the first target text and the second target text using the WMD algorithm, the cosine distance between the first target word vector and the second target word vector is calculated as the transfer cost, the TF-IDF value is used for weight allocation, and the minimum value of the sum of the products of the transfer cost and the allocation weight is used as the distance between the first target text and the second target text, and the distance is the similarity of the distance between the first target text and the second target text.

[0089] In this scheme, the algorithm's operating efficiency is improved by calculating the transfer cost between word vectors and then using the transfer cost and weight to calculate the similarity of the distance between texts. TF-IDF is also used as the weight for weight allocation in WMD, that is, the TF-IDF&Word2Vec text representation model is used as the input of the WMD algorithm, and TF-IDF&Word2Vec is used to vectorize the text, which objectively reflects the differences between words and avoids the excessive influence of high-frequency words on the document itself.

[0090] In the Word2vec model, words with similar semantics are close in distance in the embedded space, that is, the word vectors are similar. From a semantic point of view, it can be determined that for word w, among the tens of thousands of words in the corpus, only a few words are closely related, that is, a small number of words are close to word w in spatial distance, and the rest are relatively far away. Therefore, multiple words can be divided according to this distance relationship. In the specific implementation process, when using the WMD algorithm to calculate the similarity between the above-mentioned first target text and the second target text, the cosine distance between the above-mentioned first target word vector and the above-mentioned second target word vector is calculated as the transfer cost, and the TF-IDF value is used for weight allocation. Before using the minimum value of the sum of the products of the transfer cost and the allocation weight as the distance between the above-mentioned first target text and the above-mentioned second target text, the above method It also includes the following steps: obtaining a corpus, wherein the corpus includes multiple dictionary words; selecting one of the dictionary words from the corpus in turn as the central dictionary word; respectively calculating the cosine distance between the non-central dictionary words in the corpus and the central dictionary words, wherein the cosine distance is the cosine value of the angle between the word vector corresponding to the non-central dictionary word in the corpus and the word vector corresponding to the central dictionary word; storing the non-central dictionary words whose cosine distance is within the target range in the unrelated set of the central dictionary words, and storing the non-central dictionary words whose cosine distance is not within the target range in the related set of the central dictionary words.

[0091] In this scheme, the non-central dictionary words corresponding to the central dictionary words can be divided first, and the non-central dictionary words can be divided according to the cosine distance. In this way, after multiple non-central dictionary words are divided into different sets, the similarity between the non-central dictionary words in different sets and the central dictionary words can be calculated subsequently.

[0092] For example, if the order of words is based on their distance from the word vector, for the word "region A", "city B" and "Chinese" should be ranked before "guitar" and "cup". It is not considered which word "guitar" or "cup" is farther away from "region A" because they have no obvious semantic relationship with the word "region A" to be compared. Randomly select six words, namely "world, weekend, region A, family, culture, entertainment" as the central dictionary words and calculate the cosine distance in the embedding space with the non-central dictionary words in the corpus, that is, the word transfer cost c(i,j) in WMD. And sort the results from small to large, such as Figure 6 For these six non-central dictionary words, only a few words are close to each other. Taking the inflection point as the boundary, the distance between most words and the central dictionary words falls in the interval [0.6, 0.8].

[0093] To avoid contingency, 1000 central dictionary words are randomly selected to calculate their cosine distance with non-central dictionary words in the vector space, which is the transfer cost c(i,j) mentioned above. Figure 7 The distance distribution between words is shown. It can be seen that the distances between the remaining words and the central dictionary words are highly concentrated around the interval [0.6, 0.8], which is similar to the normal distribution.

[0094] In order to further improve the efficiency and accuracy of the WMD algorithm, the words can be divided into relevant sets and irrelevant sets according to their distribution in the embedding space. In some embodiments, in the process of using the WMD algorithm to calculate the similarity between the first target text and the second target text, the cosine distance between the first target word vector and the second target word vector is calculated as the transfer cost. Before using the TF-IDF value for weight distribution, the method further includes: in the case that some of the second target word vectors in the second target set are in the relevant set of the first target word vector, and some of the second target word vectors are in the irrelevant set of the first target word vector, the cosine distance between the first target word vector and the second target word vector is calculated. The cosine distance of the above-mentioned second target word vector in the relevant set is used as the transfer cost of the weight allocation of the above-mentioned second target word vector, and the TF-IDF value is used for weight allocation; the second average value of the distance from the above-mentioned second target word vector to the above-mentioned first target word vector in the above-mentioned irrelevant set of the above-mentioned first target word vector is calculated as the transfer cost of the weight allocation of the above-mentioned second target word vector, and the TF-IDF value is used for weight allocation; the sum of the first data and the second data is calculated to obtain the WMD distance between the above-mentioned first target text and the above-mentioned second target text, the above-mentioned first data is the minimum value of the sum of the products of multiple cosine distances and the weight of the above-mentioned second target word vector, and the above-mentioned second data is the product of the above-mentioned second average value and the weight of the above-mentioned second target word vector.

[0095] In this scheme, the similarity between texts can be calculated based on the WMD algorithm. By determining the distribution of words in the embedding space, the words are divided into related sets and unrelated sets. The words in the related sets and unrelated sets are combined to calculate the WMD distance. This greatly reduces the number of calculations of the distance between words and the weight distribution, reduces the calculation time of the algorithm, improves the running efficiency of the algorithm, and thus improves the efficiency of text clustering.

[0096] Specifically, based on Figure 6 and Figure 7 In order to simplify the WMD calculation process, this solution designs a new WMD algorithm, which is described as follows:

[0097] Given a word, in the corpus dictionary, except for word w, the remaining words can be divided into two groups RE(w) and URE(w). RE(w) includes words related to word w, and URE(w) includes words unrelated to w; the distance between each word in URE(w) and w is averaged, and C is used avg To represent. The distance between all words in the irrelevant set and w is approximated by C avg replace.

[0098] Combine Figure 6 It can be seen that, with the number of words corresponding to the inflection point as the limit, the number of words in RE(w) should be less than the number corresponding to the inflection point, and the number of words in URE(w) should be greater than the number corresponding to the inflection point. Figure 7 It can be determined that the distances between words in the irrelevant set and w are roughly normally distributed, so it is reasonable enough to use the mean distance in the irrelevant word set to represent the transfer cost of the word in this scheme.

[0099] Use the parameter r(w) to determine the number of RE(w) words in the relevance set of word w. Arrange the remaining words in ascending order of their distance from the embedding space of w, take the first r(w) words and place them in the relevance set RE(w), and put the rest into URE(w). For the convenience of referring to the improved WMD algorithm below, the improved WMD algorithm is named RE-WMD. The formula for calculating the word shift cost when using RE-WMD to calculate the distance between documents D and D' is:

[0100]

[0101] Where C is the RE(w) set, which includes all related words of the word. The above proposal is that for any word w in the corpus, the remaining words are divided into related sets and irrelevant sets according to their distance from the word. The transfer cost between the words in the irrelevant set and w is the average of all distances C avg Instead. This scheme is applied to WMD to improve the efficiency of the algorithm. The key idea of ​​the RE-WMD algorithm is to pre-judge whether the words w and w' to be calculated exist in the relevant set C during the document similarity calculation process. If so, calculate the cosine distance as the transfer cost. Otherwise, directly use C avg As the transfer cost between two words. Figure 6 and Figure 7 It is certain that only some words need to be calculated for cosine distance, which can avoid a lot of calculations and improve the efficiency of the algorithm. Therefore, the calculation formula of RE-WMD is:

[0102]

[0103] Specifically, this solution uses an improved Word Shift Distance (WMD) algorithm, replacing the traditional distance similarity metric as the clustering criterion. Both Euclidean and cosine distances ignore the impact of individual words on documents, resulting in inaccurate calculations and impacting subsequent clustering. Therefore, this method uses the Word Shift Distance (WMD) algorithm to improve calculation accuracy.

[0104] In one embodiment of the present application, clustering the target texts according to the similarity includes: when using one target text for the first clustering, using one target text as the first text cluster; when using the Nth target text for the Nth clustering, comparing the similarity of the Nth target text with each target text in the formed text cluster, where N≥2; when the similarity is greater than or equal to a similarity threshold, classifying the Nth target text into the text cluster; when the similarity is less than the similarity threshold, creating a new text cluster based on the Nth target text.

[0105] In this scheme, if it is the first clustering, since there are no text clusters, the first target text can be used as the first text cluster, and the subsequent multiple text clusters are also a target text as a text cluster. When processing the Nth target text, the Nth target text is classified into the text cluster with the closest similarity, that is, the text cluster with a similarity greater than the similarity threshold. If the similarity is low, then the Nth target text is used as a new text cluster. In this way, since the Word2vec&TF-IDF-WMD algorithm is used to calculate the similarity at the beginning, the obtained similarity is relatively accurate, thereby ensuring that the clustering accuracy in this embodiment is also high.

[0106] Specifically, the incremental clustering Single-Pass algorithm can be used to cluster multiple target texts. The Single-Pass algorithm is also called the single-channel algorithm and is widely used in text clustering. The Single-Pass algorithm is an efficient and simple unsupervised clustering algorithm, such as Figure 8 As shown, the algorithm process is as follows:

[0107] The clustering algorithm starts to calculate and obtains multiple n target texts after preprocessing. The n target texts constitute the corpus M, M={m1,m2,m3,...m n}, when clustering, because the initial algorithm does not generate any text clusters, that is, to determine whether there are already generated text clusters, if not, then the first target text m1 input is used as the first text cluster, that is, to create a text cluster, m1 is used as the center of the first text cluster, if there is a text cluster, when processing the i-th text sentence, m1 is used as the center of the first text cluster.i Compare the similarity with each target text in the above text cluster, that is, calculate m i and m j Similarity between them, find the sentence m with the largest similarity from the i-1th target text k , and determine whether the similarity is greater than the similarity threshold T C If the similarity is greater than the threshold, it is classified into the corresponding text cluster, otherwise a new text cluster is created. The above process is repeated until all target texts have participated in the clustering and there is no unprocessed text in the corpus, and the clustering process ends.

[0108] The embodiments of the present application also provide a text clustering device. It should be noted that the text clustering device of the embodiments of the present application can be used to execute the text clustering method provided by the embodiments of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0109] The text clustering device provided in the embodiments of the present application is introduced below.

[0110] Figure 9 FIG. 1 is a structural block diagram of a text clustering device according to an embodiment of the present application. Figure 9 As shown, the device includes:

[0111] A first acquisition unit 10 is configured to acquire a plurality of original texts and preprocess each of the original texts to obtain a plurality of target texts, wherein the preprocessing includes at least one of the following: word segmentation processing and stop word removal processing, and the target texts include a plurality of words;

[0112] A first processing unit 20 is configured to convert the words in each target text into word vectors using a Word2vec model, and determine weights of the words using a TF-IDF algorithm, wherein the weights represent the importance of the words in the target text.

[0113] The second processing unit 30 is configured to use the word vectors and the weight values ​​as inputs of a WMD algorithm, determine the similarity between any two target texts, and cluster the target texts according to the similarity.

[0114] Through this embodiment, the word vector is represented by combining the Word2vec model with the TF-IDF algorithm to enhance the distinction between different texts. The advantages of word vectors are utilized while the influence of words on texts is also taken into account. The word vector represented by the combination of the two is used as the input of the WMD algorithm. The WMD algorithm is used as the similarity measurement algorithm in text clustering to improve the accuracy of text clustering.

[0115] After the original text has been preprocessed, the data set of the original text will represent multiple words, that is, the target text. In order to effectively apply the set of target texts with this structure to subsequent clustering analysis, the text needs to be vectorized. In the specific implementation process, the first processing unit includes a construction module, a first processing module, an acquisition module and a first determination module. The construction module is used to construct a Word2vec model, wherein the above-mentioned Word2vec model is trained using multiple sets of training data, and each set of training data in the above-mentioned multiple sets of training data includes: historical words and historical word vectors corresponding to the above-mentioned historical words, and different sets of the above-mentioned training data are obtained in different historical time periods; The first processing module is used to input the above-mentioned words into the above-mentioned Word2vec model, and the output of the above-mentioned Word2vec model is the above-mentioned word vector corresponding to the above-mentioned words; the acquisition module is used to obtain the word frequency and inverse document frequency of the above-mentioned words in the above-mentioned target text, and the above-mentioned inverse document frequency is used to characterize the importance of the above-mentioned words; the first determination module is used to adopt the TF-IDF algorithm to determine the above-mentioned weight value of the above-mentioned words according to the above-mentioned word frequency and the above-mentioned inverse document frequency, wherein, the smaller the product of the above-mentioned word frequency and the above-mentioned inverse document frequency, the smaller the above-mentioned weight value of the above-mentioned words, and the lower the degree of importance, and the larger the product of the above-mentioned word frequency and the above-mentioned inverse document frequency, the larger the above-mentioned weight value of the above-mentioned words, and the higher the degree of importance.

[0116] In this solution, the Word2vec model is used to convert the words in the target text into word vectors of the same dimension. The weights of the words in the target text can also be calculated, so that the target text can be represented using word vectors and weight values. The TF-IDF algorithm is used to weight the word vectors obtained by the Word2vec model. TF-IDF is then used as the weight for weight allocation in WMD. That is, the TF-IDF & Word2Vec text representation model is used as the input to the WMD distance similarity algorithm. This objectively reflects the differences between words, avoids the excessive influence of high-frequency words on the document itself, and further improves the accuracy of text clustering.

[0117] In order to further ensure that the weight values ​​of the words obtained using the TF-IDF algorithm are more accurate, the first processing unit includes a second determination module, which is used to determine the weight values ​​of the words according to the target formula:

[0118]

[0119] Determine the weight value of the above word, where f represents the above word, m represents the above target text, W(f,m) represents the weight value of the above word f in the above target text m, TF(f,m) represents the number of times the above word f appears in the above target text m, N represents the total number of the above target text, and n i represents the number of target texts m containing the word f.

[0120] In this solution, the initial formula of the TF-IDF algorithm can be further normalized to obtain a target formula, so as to further use the target formula to ensure that the weight values ​​of the obtained words are more accurate.

[0121] In order to further improve the efficiency and accuracy of the WMD algorithm, the words can be divided into relevant sets and irrelevant sets according to their distribution in the embedding space. In some embodiments, the second processing unit includes a second processing module, a third processing module and a fourth processing module. The second processing module is used to perform TF-IDF&Word2vec vectorization on the first target text according to multiple first word vectors in the first target text and the first TF-IDF value corresponding to each of the above first word vectors to obtain a first target set, wherein the first target set includes multiple first target word vectors and weight values, and the above first word vectors and the above first TF-IDF value serve as the input of the above WMD algorithm; the third processing module is used to perform TF-IDF&Word2vec vectorization on the first target text according to multiple second word vectors in the second target text and the first TF-IDF value corresponding to each of the above second word vectors The corresponding second TF-IDF value is used to perform TF-IDF&Word2vec vectorization on the above-mentioned second target text to obtain a second target set, wherein the above-mentioned second target set includes multiple second target word vectors and weight values, and the above-mentioned second word vectors and the above-mentioned second TF-IDF value are used as inputs of the above-mentioned WMD algorithm; the fourth processing module is used to calculate the cosine distance between the above-mentioned first target word vector and the above-mentioned second target word vector as the transfer cost in the process of calculating the similarity between the above-mentioned first target text and the second target text using the WMD algorithm, use the TF-IDF value for weight allocation, and use the minimum value of the sum of the products of the transfer cost and the allocation weight as the distance between the above-mentioned first target text and the above-mentioned second target text, and the above-mentioned distance is the similarity of the distance between the above-mentioned first target text and the above-mentioned second target text.

[0122] In this scheme, the algorithm's operating efficiency is improved by calculating the transfer cost between word vectors and then using the transfer cost and weight to calculate the similarity of the distance between texts. TF-IDF is also used as the weight for weight allocation in WMD, that is, the TF-IDF&Word2Vec text representation model is used as the input of the WMD algorithm, and TF-IDF&Word2Vec is used to vectorize the text, which objectively reflects the differences between words and avoids the excessive influence of high-frequency words on the document itself.

[0123] In the Word2vec model, semantically similar words are close in distance in the embedded space, that is, the word vectors are similar. From a semantic point of view, it can be determined that for word w, among the tens of thousands of words in the corpus, only a few words are closely related, that is, a small number of words are close to word w in spatial distance, and the rest are relatively far away. Therefore, multiple words can be divided according to this distance relationship. In the specific implementation process, the above-mentioned device also includes a second acquisition unit, a selection unit, a calculation unit and a storage unit. The second acquisition unit is used to calculate the cosine distance between the above-mentioned first target word vector and the above-mentioned second target word vector as the transfer cost in the process of calculating the similarity of the above-mentioned first target text and the second target text using the WMD algorithm, use the TF-IDF value for weight allocation, and use the minimum value of the sum of the products of the transfer cost and the allocation weight as the transfer cost between the above-mentioned first target text and the above-mentioned second target text. Before calculating the distance between the two target texts, a corpus is obtained, wherein the corpus includes multiple dictionary words; the selection unit is used to select one of the above dictionary words from the corpus in turn as the central dictionary word; the calculation unit is used to respectively calculate the cosine distance between the non-central dictionary words in the corpus and the above central dictionary words, wherein the cosine distance is the cosine value of the angle between the word vector corresponding to the non-central dictionary word in the corpus and the word vector corresponding to the above central dictionary word; the storage unit is used to store the above non-central dictionary words whose cosine distance is within the target range in an unrelated set of the above central dictionary words, and store the above non-central dictionary words whose cosine distance is not within the target range in a related set of the above central dictionary words.

[0124] In this scheme, the non-central dictionary words corresponding to the central dictionary words can be divided first, and the non-central dictionary words can be divided according to the cosine distance. In this way, after multiple non-central dictionary words are divided into different sets, the similarity between the non-central dictionary words in different sets and the central dictionary words can be calculated subsequently.

[0125] In order to further improve the efficiency and accuracy of the WMD algorithm, the words can be divided into relevant sets and irrelevant sets according to their distribution in the embedding space. In some embodiments, the above-mentioned device also includes a third processing unit, which is used to calculate the cosine distance between the above-mentioned first target word vector and the above-mentioned second target word vector as the transfer cost in the process of calculating the similarity of the above-mentioned first target text and the second target text using the WMD algorithm. Before using the TF-IDF value for weight allocation, the above-mentioned method also includes: in the case that some of the above-mentioned second target word vectors in the above-mentioned second target set are in the above-mentioned relevant set of the above-mentioned first target word vector, and some of the above-mentioned second target word vectors are in the above-mentioned irrelevant set of the above-mentioned first target word vector, calculating the above-mentioned The cosine distance of the above-mentioned second target word vector in the above-mentioned relevant set of the first target word vector is used as the transfer cost of the weight allocation of the above-mentioned second target word vector, and the TF-IDF value is used for weight allocation; the second average value of the distance from the above-mentioned second target word vector to the above-mentioned first target word vector in the above-mentioned irrelevant set of the above-mentioned first target word vector is calculated as the transfer cost of the weight allocation of the above-mentioned second target word vector, and the TF-IDF value is used for weight allocation; the sum of the first data and the second data is calculated to obtain the WMD distance between the above-mentioned first target text and the above-mentioned second target text, the above-mentioned first data is the minimum value of the sum of the products of multiple cosine distances and the weight of the above-mentioned second target word vector, and the above-mentioned second data is the product of the above-mentioned second average value and the weight of the above-mentioned second target word vector.

[0126] In this scheme, the similarity between texts can be calculated based on the WMD algorithm. By determining the distribution of words in the embedding space, the words are divided into related sets and unrelated sets. The words in the related sets and unrelated sets are combined to calculate the WMD distance. This greatly reduces the number of calculations of the distance between words and the weight distribution, reduces the calculation time of the algorithm, improves the running efficiency of the algorithm, and thus improves the efficiency of text clustering.

[0127] In one embodiment of the present application, the second processing unit includes a fifth processing module, a sixth processing module and a seventh processing module. The fifth processing module is used to use the above-mentioned target text as the first text cluster when using the above-mentioned target text for the first clustering; the sixth processing module is used to compare the similarity between the Nth target text and each of the above-mentioned target texts in the above-mentioned text clusters when using the Nth target text for the Nth clustering, where N≥2; the seventh processing module is used to classify the Nth target text into the above-mentioned text cluster when the above-mentioned similarity is greater than or equal to the similarity threshold, and to create a new above-mentioned text cluster based on the Nth target text when the above-mentioned similarity is less than the above-mentioned similarity threshold.

[0128] In this scheme, if it is the first clustering, since there are no text clusters, the first target text can be used as the first text cluster, and the subsequent multiple text clusters are also a target text as a text cluster. When processing the Nth target text, the Nth target text is classified into the text cluster with the closest similarity, that is, the text cluster with a similarity greater than the similarity threshold. If the similarity is low, then the Nth target text is used as a new text cluster. In this way, since the Word2vec&TF-IDF-WMD algorithm is used to calculate the similarity at the beginning, the obtained similarity is relatively accurate, thereby ensuring that the clustering accuracy in this embodiment is also high.

[0129] The text clustering device includes a processor and a memory. The first acquisition unit, the first processing unit, and the second processing unit are all stored as program units in the memory. The processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0130] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to accurately cluster the text.

[0131] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0132] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the text clustering method.

[0133] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the text clustering method when running.

[0134] An embodiment of the present invention provides a device comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of a text clustering method are implemented. The device herein may be a server, a PC, a PAD, a mobile phone, or the like.

[0135] The present application also provides a computer program product which, when executed on a data processing device, is adapted to execute the steps of a program for initializing a clustering method with at least text.

[0136] The present application also provides a text clustering system, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any of the above-mentioned text clustering methods.

[0137] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0138] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1The function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0142] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0143] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0144] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0145] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0146] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0147] 1) The text clustering method of this application uses a combination of the Word2vec model and the TF-IDF algorithm to represent word vectors, thereby enhancing the distinction between different texts. It not only takes advantage of the advantages of word vectors but also incorporates the influence of words on text. The word vectors represented by the combination of the two are used as the input of the WMD algorithm. The WMD algorithm is used as the similarity measurement algorithm in text clustering to improve the accuracy of text clustering.

[0148] 2) The text clustering device of the present application uses a combination of the Word2vec model and the TF-IDF algorithm to represent word vectors, thereby enhancing the distinction between different texts. It not only utilizes the advantages of word vectors but also adds the influence of words on texts. The word vector represented by the combination of the two is used as the input of the WMD algorithm. The WMD algorithm is used as the similarity measurement algorithm in text clustering to improve the accuracy of text clustering.

[0149] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A text clustering method, characterized in that: include: Acquire multiple original texts, perform preprocessing on each of the original texts to obtain multiple target texts, wherein the preprocessing includes at least one of the following: word segmentation processing and stop word removal processing, and the target text includes multiple words; The words in each target text are converted into word vectors using a Word2vec model, and the weight values ​​of the words are determined using a TF-IDF algorithm, wherein the weight values ​​represent the importance of the words in the target text; According to multiple first word vectors in the first target text and the first TF-IDF values ​​corresponding to each of the first word vectors, the first target text is vectorized by TF-IDF&Word2vec to obtain a first target set, wherein the first target set includes multiple first target word vectors and weight values, and the first word vectors and the first TF-IDF values ​​serve as inputs to the WMD algorithm; according to multiple second word vectors in the second target text and the second TF-IDF values ​​corresponding to each of the second word vectors, the second target text is vectorized by TF-IDF&Word2vec to obtain a second target set, wherein the second target set includes multiple second target word vectors and weight values, and the second word vectors and the second TF-IDF values ​​serve as inputs to the WMD algorithm; Acquire a corpus, wherein the corpus includes a plurality of dictionary words; sequentially select one of the dictionary words from the corpus as a central dictionary word; respectively calculate a cosine distance A between a non-central dictionary word in the corpus and the central dictionary word, wherein the cosine distance A is a cosine value of an angle between a word vector corresponding to the non-central dictionary word in the corpus and a word vector corresponding to the central dictionary word; store the non-central dictionary words whose cosine distance A is within a target range in an unrelated set of the central dictionary words, and store the non-central dictionary words whose cosine distance A is not within the target range in a related set of the central dictionary words; In the case where some of the second target word vectors in the second target set are in the related set of the first target word vector, and some of the second target word vectors are in the irrelevant set of the first target word vector, calculate the cosine distance B of the second target word vector in the related set of the first target word vector as the transfer cost of the weight assignment of the second target word vector, and use the TF-IDF value for weight assignment; calculate the second average value of the distance from the second target word vector in the irrelevant set of the first target word vector to the first target word vector as the transfer cost of the weight assignment of the second target word vector, and use the TF-IDF value for weight assignment; calculate the sum of the first data and the second data to obtain the WMD distance between the first target text and the second target text, the first data being the minimum value of the sum of multiple products of the cosine distance B and the weight of the second target word vector, and the second data being the product of the second average value and the weight of the second target word vector; In the process of calculating the similarity between the first target text and the second target text using the WMD algorithm, the cosine distance between the first target word vector and the second target word vector is calculated as the transfer cost, the TF-IDF value is used for weight assignment, and the minimum value of the sum of the products of the transfer cost and the assigned weight is used as the distance between the first target text and the second target text. The distance is the similarity of the distance between the first target text and the second target text, and the target texts are clustered according to the similarity.

2. The method according to claim 1, characterized in that The words in each target text are converted into word vectors using the Word2vec model, and the weight values ​​of the words are determined using the TF-IDF algorithm, including: Constructing a Word2vec model, wherein the Word2vec model is trained using multiple sets of training data, each set of training data including: historical words and historical word vectors corresponding to the historical words, and different sets of training data are acquired in different historical time periods; Input the word into the Word2vec model, and obtain the output of the Word2vec model as the word vector corresponding to the word; Obtaining the word frequency and inverse document frequency of the word in the target text, wherein the inverse document frequency is used to represent the importance of the word; The TF-IDF algorithm is adopted to determine the weight value of the word according to the word frequency and the inverse document frequency, wherein the smaller the product of the word frequency and the inverse document frequency is, the smaller the weight value of the word is and the lower the importance is; the larger the product of the word frequency and the inverse document frequency is, the larger the weight value of the word is and the higher the importance is.

3. The method according to claim 1, characterized in that The TF-IDF algorithm is used to determine the weight value of the word, and further includes: According to the target formula: ； Determine the weight value of the word, wherein, Indicates the word represents the target text, Indicates the word In the target text The weight value in Indicates the word In the target text The number of times it appears in Indicates the total number of target texts, Indicates that it contains the words The target text The number of .

4. The method according to claim 1, wherein Clustering the target text according to the similarity includes: In the case of using one target text to perform the first clustering, taking one target text as the first text cluster; When the Nth target text is used to perform the Nth clustering, the Nth target text is compared with each target text in the formed text clusters for similarity, wherein N≥2; When the similarity is greater than or equal to a similarity threshold, the Nth target text is classified into the text cluster; when the similarity is less than the similarity threshold, a new text cluster is created based on the Nth target text.

5. A text clustering device, characterized in that: include: A first acquisition unit is configured to acquire a plurality of original texts and preprocess each of the original texts to obtain a plurality of target texts, wherein the preprocessing includes at least one of the following: word segmentation processing and stop word removal processing, and the target texts include a plurality of words; A first processing unit is configured to convert the words in each target text into word vectors using a Word2vec model, and determine weight values ​​of the words using a TF-IDF algorithm, wherein the weight values ​​represent the importance of the words in the target text; a second acquiring unit, configured to acquire a corpus, wherein the corpus includes a plurality of dictionary words; A selection unit, configured to sequentially select one of the dictionary words from the corpus as a central dictionary word; a calculation unit, configured to respectively calculate a cosine distance A between a non-central dictionary word and a central dictionary word in the corpus, wherein the cosine distance A is a cosine value of an angle between a word vector corresponding to a non-central dictionary word in the corpus and a word vector corresponding to the central dictionary word; a storage unit, configured to store the non-central dictionary words whose cosine distance A is within a target range into an unrelated set of central dictionary words, and store the non-central dictionary words whose cosine distance A is not within the target range into a related set of central dictionary words; A third processing unit is configured to calculate, when some second target word vectors in the second target set are in the relevant set of the first target word vector and some second target word vectors are in the irrelevant set of the first target word vector, the cosine distance B of the second target word vector in the relevant set of the first target word vector as the transfer cost for weight assignment of the second target word vector, and use the TF-IDF value for weight assignment; calculate the second average value of the distance from the second target word vector in the irrelevant set of the first target word vector to the first target word vector as the transfer cost for weight assignment of the second target word vector, and use the TF-IDF value for weight assignment; calculate the sum of first data and second data to obtain the WMD distance between the first target text and the second target text, the first data being the minimum value of the sum of multiple products of the cosine distance B and the weight of the second target word vector, and the second data being the product of the second average value and the weight of the second target word vector; a second processing unit, configured to use the word vector and the weight value as inputs of a WMD algorithm, determine the similarity between any two target texts, and cluster the target texts according to the similarity; The second processing unit includes a second processing module, a third processing module and a fourth processing module. The second processing module is used to perform TF-IDF&Word2vec vectorization on the first target text according to multiple first word vectors in the first target text and the first TF-IDF value corresponding to each of the first word vectors to obtain a first target set, wherein the first target set includes multiple first target word vectors and weight values, and the first word vectors and the first TF-IDF value serve as the input of the WMD algorithm; the third processing module is used to perform TF-IDF&Word2vec vectorization on the second target text according to multiple second word vectors in the second target text and the second TF-IDF value corresponding to each of the second word vectors Word2vec vectorization representation is used to obtain a second target set, wherein the second target set includes multiple second target word vectors and weight values, and the second word vectors and the second TF-IDF value are used as inputs of the WMD algorithm; the fourth processing module is used to use the WMD algorithm to calculate the similarity between the first target text and the second target text, calculate the cosine distance between the first target word vector and the second target word vector as the transfer cost, use the TF-IDF value for weight allocation, and use the minimum value of the sum of the products of the transfer cost and the allocation weight as the distance between the first target text and the second target text, and the distance is the similarity of the distance between the first target text and the second target text.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the text clustering method according to any one of claims 1 to 4.

7. A text clustering system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the text clustering method described in any one of claims 1 to 4.

Images