Text clustering methods, devices, computer equipment and storage media

Abstract

This invention provides a text clustering method, apparatus, computer device, and storage medium. The method includes: acquiring multiple target texts and determining a text vector for each target text; calculating a first similarity between the text vectors and dividing the multiple target texts into multiple initial categories based on the first similarity; ensuring that the first similarity between two or more text vectors belonging to the same initial category is greater than a preset first threshold; calculating a second similarity between the multiple initial categories and merging the multiple initial categories based on the second similarity to obtain multiple final categories. This invention consists of two stages: coarse clustering and fine clustering. In the coarse clustering stage, texts with similarity higher than a certain large threshold are clustered together to ensure clustering accuracy. In the fine clustering stage, the two clusters with the closest cluster centers are continuously merged until the overall clustering degree no longer increases, ensuring the final clustering effect while achieving rapid clustering.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a text clustering method, apparatus, computer device, and storage medium. Background Technology

[0002] Text clustering is the process of grouping multiple text sentences into different clusters based on their similarity, so that sentences with high similarity belong to the same cluster, and sentences with low similarity belong to different clusters. Currently, common implementation methods include cluster-center-based algorithms (k-means), bottom-up clustering algorithms (hierarchical clustering), and density clustering. However, existing text clustering methods all have some inherent drawbacks. For example, the performance of cluster-center-based algorithms heavily depends on the selection of the number of clusters and the cluster centers, while hierarchical clustering and density clustering require long computation times. Therefore, how to provide an algorithm that can quickly and accurately implement text clustering has become a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0003] The purpose of this invention is to provide a technical solution that can optimize text clustering results and improve text clustering efficiency, so as to solve the above-mentioned problems existing in the prior art.

[0004] To achieve the above objectives, the present invention provides a text clustering method, comprising the following steps:

[0005] Obtain multiple target texts and determine the text vector for each target text;

[0006] Calculate the first similarity between the text vectors, and divide the multiple target texts into multiple initial categories based on the first similarity; wherein the first similarity between two or more text vectors belonging to the same initial category is greater than a preset first threshold;

[0007] Calculate a second similarity among the multiple initial categories, and merge the multiple initial categories based on the second similarity to obtain multiple final categories.

[0008] According to the text clustering method provided by the present invention, the step of obtaining multiple target texts and determining the text vector of each target text includes:

[0009] The target text is input into a trained language processing model to output a text vector corresponding to the target text.

[0010] According to the text clustering method provided by the present invention, the step of dividing the plurality of target texts into a plurality of initial categories based on the first similarity includes:

[0011] Get any first target text, and determine whether the first similarity between any other second target text and the first target text is greater than or equal to a first threshold. If yes, add the first target text and the second target text to the first initial category. If no, continue to determine the first similarity between any other third target text and the first target text, until all target texts are traversed.

[0012] Obtain any unclassified fourth target text, and determine whether the first similarity between any unclassified fifth target text and the fourth target text is greater than or equal to a first threshold. If yes, add the fourth target text and the fifth target text to the second initial category; otherwise, continue to determine the first similarity between any unclassified sixth target text and the fourth target text, until all unclassified target texts are traversed.

[0013] According to the text clustering method provided by the present invention, the step of calculating a second similarity among multiple initial categories and merging the multiple initial categories based on the second similarity to obtain multiple final categories includes:

[0014] Calculate the center vector value of each initial category and the second similarity between any two center vector values;

[0015] Determine the first and second initial categories with the highest second similarity;

[0016] Calculate the first clustering degree between the first initial category and the second initial category and other reference initial categories before merging, and the second clustering degree between the first initial category and the second initial category and other reference initial categories after merging;

[0017] The relationship between the second clustering degree and the first clustering degree determines whether the first initial category and the second initial category need to be merged.

[0018] According to the text clustering method provided by the present invention, the other reference initial categories include N other initial categories that are similar to the first initial category and N other initial categories that are similar to the second initial category, where N is a natural number.

[0019] According to the text clustering method provided by the present invention, the degree of clustering is determined by the ratio between the intra-class average distance and the inter-class average distance. The intra-class average distance is related to the average value of all text vectors within the same initial category, and the inter-class average distance is related to the average value of the difference between the center vector values ​​of any two initial categories.

[0020] According to the text clustering method provided by the present invention, the step of calculating a second similarity among multiple initial categories and merging the multiple initial categories based on the second similarity to obtain multiple final categories further includes:

[0021] If it is determined that the first initial category and the second initial category need to be merged, all target text contained in the first initial category and the second initial category are merged to obtain a new initial category;

[0022] Repeat the step of merging the plurality of initial categories based on the second similarity until it is determined that the first initial category and the second initial category need to be merged;

[0023] Use all current categories as the ultimate category.

[0024] To achieve the above objectives, the present invention also proposes a text clustering device, comprising:

[0025] The text acquisition module is suitable for acquiring multiple target texts and determining the text vector of each target text;

[0026] The initial category module is adapted to calculate the first similarity between the text vectors and divide the multiple target texts into multiple initial categories based on the first similarity; wherein the first similarity between two or more text vectors belonging to the same initial category is greater than a preset first threshold.

[0027] The final category module is adapted to calculate a second similarity between multiple initial categories and merge the multiple initial categories based on the second similarity to obtain multiple final categories.

[0028] To achieve the above objectives, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0029] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0030] The text clustering method, apparatus, computer device, and storage medium provided by this invention are divided into two stages based on a first similarity and a second similarity: coarse clustering and fine clustering. In the coarse clustering stage, text vectors are determined based on the target text, and then texts with similarity scores higher than a certain large threshold are clustered together using the first similarity score, thus initially ensuring the accuracy of clustering. In the fine clustering stage, the two clusters with the closest cluster centers are continuously merged until the overall clustering degree no longer increases. Through fine clustering, the final clustering effect is guaranteed while maintaining rapid clustering. This ensures both clustering effectiveness and improved clustering efficiency. Attached Figure Description

[0031] Figure 1 This is a flowchart of an embodiment of the text clustering method of the present invention;

[0032] Figure 2 This is a schematic flowchart illustrating the initial category division in Embodiment 1 of the present invention;

[0033] Figure 3 This is a schematic flowchart illustrating the merging of initial categories into final categories according to Embodiment 1 of the present invention;

[0034] Figure 4 This is a schematic diagram of the program modules of a text clustering device according to a first embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the hardware structure of a text clustering device according to a first embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0037] Example 1

[0038] Please see Figure 1 This embodiment proposes a text clustering method, including the following steps:

[0039] S100: Obtain multiple target texts and determine the text vector of each target text.

[0040] The target text is the text to be clustered, such as multiple sentences in an article or multiple words in a paragraph. Existing arbitrary vector conversion algorithms can be used to convert text into vectors, such as the word2vec method, the BERT model method, and the CNN model method; this embodiment does not impose any limitations on this. In one example, the conversion from target text to vectors can be achieved through a language processing model trained on data. This language processing model can be constructed based on an artificial neural network. Historical text samples and labeled historical vector samples are used as the input and output data of the artificial neural network, respectively, to train the coefficients of each neuron in the artificial neural network. This allows the language processing model to output a corresponding text vector for any input text.

[0041] S200: Calculate the first similarity between the text vectors, and divide the multiple target texts into multiple initial categories based on the first similarity; wherein the first similarity between two or more text vectors belonging to the same initial category is greater than a preset first threshold.

[0042] The first similarity can include the Euclidean distance or cosine similarity between any two vectors; this embodiment is not limited to this. A relatively high first threshold can be set, such as 0.95, and the initial categories are determined by comparing the relationship between the first similarity between any two text vectors and the first threshold. If the first similarity between two text vectors is greater than or equal to the first threshold, then these two text vectors are assigned to the same initial category; if the first similarity between a text vector and any other text vector is less than the first threshold, then that text vector is assigned to a separate initial category. For example, the target text includes 10 texts: text 1, text 2, ..., text 10, resulting in 10 text vectors: vector 1, vector 2, ..., vector 10. If only the first similarity between vector 2 and vector 3 is greater than the first threshold, and the first similarity between any pair of the remaining vectors is less than the first threshold, then vector 2 and vector 3 are assigned to the same initial category, and the remaining vectors are each assigned to a separate initial category, thus resulting in 9 initial categories.

[0043] By setting a high first threshold, such as a threshold higher than 0.90, it can be ensured that the similarity of target texts contained in the same initial category is high enough, and the similarity of target texts contained in different initial categories can also have relatively similar features under certain circumstances, thus providing a basis for further category merging.

[0044] Figure 2 This is a schematic flowchart illustrating the initial category division process according to Embodiment 1 of the present invention. Figure 2 As shown, step S200 includes:

[0045] S210: Obtain any first target text, determine whether the first similarity between any other second target text and the first target text is greater than or equal to a first threshold. If yes, add the first target text and the second target text to the first initial category. If no, continue to determine the first similarity between any other third target text and the first target text, until all target texts are traversed.

[0046] S220: Obtain any unclassified fourth target text, determine whether the first similarity between any unclassified fifth target text and the fourth target text is greater than or equal to a first threshold. If yes, add the fourth target text and the fifth target text to the second initial category. If no, continue to determine the first similarity between any unclassified sixth target text and the fourth target text, until all unclassified target texts are traversed.

[0047] Specifically, starting from the first text, proceed sequentially to find the first unclassified text as the new initial category and its corresponding vector as the sentinel vector. Then, starting from the vector corresponding to the first text after that, continue until the m-th vector. If the text already belongs to an initial category, skip it; otherwise, calculate the cosine similarity between the vector and the sentinel vector. If it is greater than a preset threshold, assign the text to the initial category of the sentinel vector text.

[0048] S300: Calculate a second similarity among the multiple initial categories, and merge the multiple initial categories according to the second similarity to obtain multiple final categories.

[0049] The second similarity in this embodiment may include the Euclidean distance or cosine similarity between the center vector values ​​of any two initial categories. The center vector value refers to the center value of all text vectors in the initial category. For example, if an initial category contains only one text vector, then the center vector value of the initial category is that text vector value; if an initial category contains multiple text vectors, then the center vector value of the initial category is the average of the multiple text vectors.

[0050] Figure 3 This is a schematic flowchart illustrating the merging of initial categories into final categories according to Embodiment 1 of the present invention. Figure 3 As shown, step S300 includes:

[0051] S310: Calculate the center vector value of each initial category and the second similarity between any two center vector values. As mentioned earlier, the second similarity may include the Euclidean distance or cosine similarity between the center vector values ​​of any two initial categories.

[0052] S320: Determine the first and second initial categories with the highest second similarity. It can be understood that the highest second similarity indicates that the text vectors contained in the first and second initial categories are most similar, therefore, the first and second initial categories have the potential to be merged.

[0053] S330: Calculate the first clustering degree between the first initial category and the second initial category and other reference initial categories before merging, and the second clustering degree between the first initial category and the second initial category and other reference initial categories after merging.

[0054] Clustering degree is used to measure the quality of clustering. In this step, the overall clustering degree between the first and second initial categories and other initial categories before and after merging is calculated to determine whether the first and second initial categories need to be merged. It should be noted that in this embodiment, when calculating the clustering degree, it is not necessary to calculate the clustering degree of all categories. Only the N (N is a natural number) other initial categories that are adjacent to the initial category to be merged are selected for calculation, thereby reducing the calculation process and improving the efficiency of cluster merging.

[0055] Assume there are 9 initial categories: Initial Category 1, Initial Category 2... Initial Category 9. Initial Category 1 and Initial Category 2 have the highest similarity. This step requires calculating the first clustering degree between Initial Category 1 and Initial Category 2 and their N neighboring initial categories before merging, and the second clustering degree between Initial Category 1 and Initial Category 2 and their N neighboring initial categories after merging. Assume N=2, meaning we select 2 neighboring initial categories of Initial Category 1 and 2 neighboring initial categories of Initial Category 2, for a total of 2×2=4 other initial categories. Before merging, Initial Category 1 and Initial Category 2 represent two initial categories, plus the 4 other initial categories, for a total of 6 categories. After merging, Initial Category 1 and Initial Category 2 are merged into a new category, plus the 4 other initial categories, for a total of 5 categories.

[0056] Before merging, the first clustering degree of the 2N+2 classes (2N other initial classes plus initial class 1 and initial class 2) is calculated. The first clustering degree is calculated as the average intra-class distance / average inter-class distance. To calculate the average intra-class distance, first calculate the sum of the Euclidean distances from all vectors within each initial class to the center vector, divided by the number of vectors. Then, take the average of the intra-class distances of all initial classes as the average intra-class distance. To calculate the average inter-class distance, pair any two initial classes, calculate the Euclidean distance between the centers of this pair, sum the Euclidean distances of the centers of all pairs, and divide by the total number of class pairs to obtain the average inter-class distance.

[0057] S340: Determine whether the first initial category and the second initial category need to be merged based on the relationship between the second clustering degree and the first clustering degree.

[0058] A lower clustering strength value indicates better clustering results. If the second clustering strength is lower than the first clustering strength, the merged clustering effect is better than the original clustering effect, and the first and second initial clusters are suitable for merging. If the second clustering strength is greater than the first clustering strength, the merged clustering effect is worse than the original clustering effect, and the first and second initial clusters are not suitable for merging.

[0059] Furthermore, if it is determined that the first initial category and the second initial category need to be merged, all target text contained in the first initial category and the second initial category are merged to obtain a new initial category.

[0060] Repeat the step of merging the plurality of initial categories based on the second similarity until it is determined that the first initial category and the second initial category need to be merged.

[0061] Use all current categories as the ultimate category.

[0062] In the above steps, the present invention first uses high-threshold coarse clustering, and then performs fine clustering by merging pairs. At the same time, when calculating the degree of clustering, only the clustering degree of the N nearest neighbors of the class to be merged is calculated. This method can greatly speed up the clustering process while ensuring the clustering effect.

[0063] Please continue reading. Figure 4 This paper illustrates a text clustering device. In this embodiment, the text clustering device 40 may include or be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the present invention and implement the above-described text clustering method. The program module referred to in this invention is a series of computer program instruction segments capable of performing a specific function, which is more suitable than the program itself for describing the execution process of the text clustering device 40 in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment:

[0064] The text acquisition module 41 is adapted to acquire multiple target texts and determine the text vector of each target text;

[0065] The initial category module 42 is adapted to calculate the first similarity between the text vectors and divide the multiple target texts into multiple initial categories based on the first similarity; wherein the first similarity between two or more text vectors belonging to the same initial category is greater than a preset first threshold.

[0066] The final category module 43 is adapted to calculate a second similarity between multiple initial categories and merge the multiple initial categories according to the second similarity to obtain multiple final categories.

[0067] This embodiment also provides a computer device, such as a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers), etc., capable of executing programs. The computer device 50 in this embodiment includes, but is not limited to, a memory 51 and a processor 52 that can be interconnected via a system bus. Figure 5 As shown. It should be noted that, Figure 5 Only a computer device 50 with components 51-52 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0068] In this embodiment, the memory 51 (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 51 may be an internal storage unit of the computer device 50, such as the hard disk or memory of the computer device 50. In other embodiments, the memory 51 may also be an external storage device of the computer device 50, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 50. Of course, the memory 51 may include both the internal storage unit and the external storage device of the computer device 50. In this embodiment, the memory 51 is typically used to store the operating system and various application software installed on the computer device 50, such as the program code of the text clustering device 40 in Embodiment 1. In addition, the memory 51 can also be used to temporarily store various types of data that have been output or will be output.

[0069] In some embodiments, processor 52 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 52 is typically used to control the overall operation of computer device 50. In this embodiment, processor 52 is used to run program code stored in memory 51 or process data, for example, to run text clustering device 50 to implement the text clustering method of Embodiment 1.

[0070] This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. When the program is executed by a processor, it implements the corresponding function. In this embodiment, the computer-readable storage medium is used to store the text clustering device 40, which, when executed by a processor, implements the text clustering method of Embodiment 1.

[0071] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0072] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0073] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0074] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0076] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A text clustering method, characterized in that, Includes the following steps: Obtain multiple target texts and determine the text vector for each target text; Calculate the first similarity between the text vectors, and divide the multiple target texts into multiple initial categories based on the first similarity; Among them, the first similarity between two or more text vectors belonging to the same initial category is greater than the preset first threshold; Calculate a second similarity among the multiple initial categories, and merge the multiple initial categories according to the second similarity to obtain multiple final categories; The step of calculating a second similarity among the multiple initial categories and merging the multiple initial categories based on the second similarity to obtain multiple final categories includes: Calculate the center vector value of each initial category and the second similarity between any two center vector values; Determine the first and second initial categories with the highest second similarity; Calculate the first clustering degree between the first initial category and the second initial category and other reference initial categories before merging, and the second clustering degree between the first initial category and the second initial category and other reference initial categories after merging; wherein, the clustering degree is determined by the ratio between the intra-class average distance and the inter-class average distance, wherein the intra-class average distance is related to the average value of all text vectors within the same initial category, and the inter-class average distance is related to the average difference between the center vector values ​​of any two initial categories; The relationship between the second clustering degree and the first clustering degree determines whether the first initial category and the second initial category need to be merged.

2. The text clustering method according to claim 1, characterized in that, The step of acquiring multiple target texts and determining the text vector of each target text includes: The target text is input into a trained language processing model to output a text vector corresponding to the target text.

3. The text clustering method according to claim 1 or 2, characterized in that, The step of classifying the plurality of target texts into a plurality of initial categories based on the first similarity includes: Get any first target text, and determine whether the first similarity between any other second target text and the first target text is greater than or equal to a first threshold. If yes, add the first target text and the second target text to the first initial category. If no, continue to determine the first similarity between any other third target text and the first target text, until all target texts are traversed. Obtain any unclassified fourth target text, and determine whether the first similarity between any unclassified fifth target text and the fourth target text is greater than or equal to a first threshold. If yes, add the fourth target text and the fifth target text to the second initial category; otherwise, continue to determine the first similarity between any unclassified sixth target text and the fourth target text, until all unclassified target texts are traversed.

4. The text clustering method according to claim 1, characterized in that, The other reference initial categories include N other initial categories that are similar to the first initial category and N other initial categories that are similar to the second initial category, where N is a natural number.

5. The text clustering method according to claim 1, characterized in that, The step of calculating a second similarity among the multiple initial categories and merging the multiple initial categories based on the second similarity to obtain multiple final categories further includes: If it is determined that the first initial category and the second initial category need to be merged, all target text contained in the first initial category and the second initial category are merged to obtain a new initial category; Repeat the step of merging the plurality of initial categories based on the second similarity until it is determined that the first initial category and the second initial category need to be merged; Use all current categories as the ultimate category.

6. A text clustering device, characterized in that, include: The text acquisition module is suitable for acquiring multiple target texts and determining the text vector of each target text; The initial category module is adapted to calculate the first similarity between the text vectors and classify the multiple target texts into multiple initial categories based on the first similarity; Among them, the first similarity between two or more text vectors belonging to the same initial category is greater than the preset first threshold; The final category module is adapted to calculate a second similarity between multiple initial categories, and to merge the multiple initial categories according to the second similarity to obtain multiple final categories; specifically, it calculates the center vector value of each initial category and the second similarity between any two center vector values; Determine the first and second initial categories with the highest second similarity; Calculate the first clustering degree between the first initial category and the second initial category and other reference initial categories before merging, and the second clustering degree between the first initial category and the second initial category and other reference initial categories after merging; determine whether the first initial category and the second initial category need to be merged based on the relationship between the second clustering degree and the first clustering degree; wherein, the clustering degree is determined by the ratio between the intra-class average distance and the inter-class average distance, the intra-class average distance is related to the average value of all text vectors within the same initial category, and the inter-class average distance is related to the average difference between the center vector values ​​of any two initial categories.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images