Text clustering method and device, electronic equipment and storage medium

Abstract

The invention provides a text clustering method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-clustered text data set, and constructing an LSH index of the text data set; searching in the text data set by using an LSH index to obtain a plurality of similar texts corresponding to each text; calculating a text difference degree, and screening a plurality of similar texts corresponding to each text to obtain a candidate text data set corresponding to each text; taking each text in the candidate text data set as a node, creating an edge according to the candidate text data set corresponding to each text, connecting the corresponding nodes, and constructing a graph model; a graph model algorithm is adopted to determine a plurality of maximum connected sub-graphs from a graph model, and a node set in each maximum connected sub-graph is divided into a cluster. According to the method, the texts with high literal similarity can be accurately clustered, the method is suitable for clustering large-scale text data, the calculation cost is reduced, and the clustering efficiency and real-time performance are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text clustering method, device, electronic device and storage medium. Background Art

[0002] Text clustering is an important task in the field of natural language processing. Its main purpose is to classify similar texts into the same category to facilitate information organization, management and retrieval. In some specific fields, text clustering needs to be performed based on the literal similarity of the text.

[0003] Existing text clustering methods mainly convert text into vector representations and then perform clustering using methods such as K-means and hierarchical clustering. However, vector representations are effective in capturing semantic similarity, but are not accurate enough in judging literal similarity; traditional clustering methods have high computational costs and low efficiency when processing large-scale text data, making it difficult to meet real-time requirements. Summary of the invention

[0004] The present invention provides a text clustering method, device, electronic device and storage medium, which are used to solve the defects in the prior art that the text is converted into a vector representation and then clustered by combining K-means, hierarchical clustering and other methods, the judgment of literal similarity is not accurate enough, and when processing large-scale text data, the calculation cost is high, the efficiency is low, and it is difficult to meet the real-time requirements.

[0005] In a first aspect, the present invention provides a text clustering method, comprising: Obtain a text data set to be clustered, and construct a local sensitive hash (LSH) index of the text data set; Using the LSH index, searching the text data set for texts similar to each text in the text data set to obtain a plurality of similar texts corresponding to each text; Calculating the text difference between a plurality of similar texts corresponding to each text and each text, screening the plurality of similar texts corresponding to each text, and obtaining a candidate text data set corresponding to each text; Taking each text in the candidate text data set as a node, creating an edge according to the candidate text data set corresponding to each text, connecting the corresponding nodes, and constructing a graph model; A graph model algorithm is used to determine a plurality of maximal connected subgraphs from the graph model, and a set of nodes in each maximal connected subgraph is divided into a cluster.

[0006] In some embodiments, calculating the text difference between the plurality of similar texts corresponding to each text and each text includes: Calculating the edit distance between a plurality of similar texts corresponding to each text and each text; Based on the edit distances between the multiple similar texts corresponding to each text and each text, and the length of each text, a normalized text difference between the multiple similar texts corresponding to each text and each text is obtained.

[0007] In some embodiments, the screening of the multiple similar texts corresponding to each text includes:

[0008] It is determined whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold; if so, each similar text corresponding to each text is added to the candidate text data set corresponding to each text.

[0009] In some embodiments, a graph model algorithm is used to determine a plurality of maximal connected subgraphs from the graph model, including: Using a graph model algorithm to traverse the graph model; Find a plurality of maximal connected subgraphs from the graph model.

[0010] In some embodiments, after determining a plurality of maximal connected subgraphs from the graph model using a graph model algorithm, the method further includes: The nodes in the graph model that are not added to any maximally connected subgraph are divided into a cluster.

[0011] In some embodiments, the method of taking each text in the candidate text dataset as a node, creating an edge according to the candidate text dataset corresponding to each text, connecting the corresponding nodes, and constructing a graph model includes: Taking each text in the candidate text data set as a node, obtaining a plurality of similar text pairs according to the candidate text data set corresponding to each text, creating corresponding edges to connect two nodes corresponding to each similar text pair, and constructing a graph model; The weight of the edge corresponding to each similar text pair is determined according to the text difference degree of each similar text pair.

[0012] In some embodiments, constructing a local sensitive hash (LSH) index of the text data set includes: Calculate the hash value of each text in the text data set by using a hash function; Based on the hash value of each text, generate a corresponding hash table; Based on the hash table, the LSH index is constructed.

[0013] In a second aspect, the present invention further provides a text clustering device, comprising: A first construction unit is used to obtain a text data set to be clustered and construct a local sensitive hash (LSH) index of the text data set; A retrieval unit, used to retrieve texts similar to each text in the text data set in the text data set by using the LSH index, and obtain multiple similar texts corresponding to each text; A screening unit, used for calculating the text difference between a plurality of similar texts corresponding to each text and each text, screening the plurality of similar texts corresponding to each text, and obtaining a candidate text data set corresponding to each text; A second construction unit, taking each text in the candidate text data set as a node, creating an edge according to the candidate text data set corresponding to each text, connecting the corresponding nodes, and constructing a graph model; The first clustering unit is used to determine a plurality of maximal connected subgraphs from the graph model by using a graph model algorithm, and divide the node set in each maximal connected subgraph into a cluster.

[0014] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the text clustering method described above is implemented.

[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the text clustering method described in any one of the above is implemented.

[0016] The text clustering method, device, electronic device and storage medium provided by the present invention construct an LSH index of a text data set, use the LSH index to search in the text data set to obtain multiple similar texts corresponding to each text, screen the multiple similar texts corresponding to each text based on text difference to obtain a candidate text data set corresponding to each text, use each text in the candidate text data set as a node, create edges according to the candidate text data set corresponding to each text, connect the corresponding nodes, construct a graph model, use a graph model algorithm to determine multiple maximal connected subgraphs from the graph model, divide the node set in each maximal connected subgraph into a clustering cluster, can accurately cluster texts with high literal similarity, is suitable for clustering large-scale text data, reduces computing costs, and improves clustering efficiency and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the text clustering method provided by the embodiment of the present invention; Figure 2 This is the second flow chart of the text clustering method provided by the embodiment of the present invention; Figure 3 It is a structural schematic diagram of a text clustering device provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 This is one of the flow charts of the text clustering method provided by the embodiment of the present invention. Figure 1 As shown, a text clustering method is provided, comprising the following steps: step 110, step 120, step 130, step 140 and step 150. The steps of the method flow are only a possible implementation of the present invention.

[0021] Step 110: Obtain a text data set to be clustered, and construct a local sensitive hash (LSH) index of the text data set.

[0022] Among them, Locality-Sensitive Hashing (LSH) is an approximate nearest neighbor search technology for high-dimensional data, which is suitable for similarity search of text data. The core idea of ​​LSH is to map similar data points into the same bucket through a hash function, thereby reducing the number of candidate pairs that need to be compared and improving search efficiency.

[0023] In some embodiments, constructing a locality sensitive hash (LSH) index of a text data set includes: Calculate the hash value of each text in the text data set using a hash function; Based on the hash value of each text, generate a corresponding hash table; Build an LSH index based on the hash table.

[0024] Optionally, the text data set is preprocessed to convert the text data into a form suitable for hash function processing.

[0025] Optionally, a suitable hash function is selected according to the type of text data and the measurement method of similarity.

[0026] Optionally, the parameters of the hash function are optimized and adjusted.

[0027] Step 120: Use the LSH index to search the text data set for texts similar to each text in the text data set, and obtain multiple similar texts corresponding to each text.

[0028] Optionally, each text in the text data set is taken as a query, and based on each query, a search is performed in the text data set to obtain a plurality of similar texts corresponding to each text.

[0029] For example, for a query, the LSH index is used to retrieve texts similar to the query in the text dataset, and three similar texts d1, d2, and d3 are obtained.

[0030] It can be understood that by using the LSH index to retrieve text similar to each text in the text dataset, the calculation and storage costs are reduced, the efficiency and real-time performance of the retrieval are improved, and it is suitable for large-scale datasets.

[0031] Step 130 : Calculate the text difference between the multiple similar texts corresponding to each text and each text, screen the multiple similar texts corresponding to each text, and obtain a candidate text data set corresponding to each text.

[0032] It is understandable that by calculating the text difference, multiple similar texts corresponding to each text can be screened based on the text difference, so as to obtain a candidate text data set with high similarity to each text.

[0033] In some embodiments, calculating the text difference between a plurality of similar texts corresponding to each text and each text includes: Calculate the edit distance between multiple similar texts corresponding to each text and each text; Based on the edit distances between the multiple similar texts corresponding to each text and each text, and the length of each text, a normalized text difference between the multiple similar texts corresponding to each text and each text is obtained.

[0034] The edit distance is the minimum number of single-character edit operations required to transform one string into another. These edit operations include insertion, deletion, and substitution. The edit distance is used to measure the difference between two strings.

[0035] For example, if the similar texts corresponding to the query are d1, d2, and d3, then the three edit distances of (query, d1), (query, d2), and (query, d3) are calculated respectively.

[0036] Optionally, the edit distance is divided by the query length to obtain a normalized text difference.

[0037] It can be understood that calculating text differences through edit distance is helpful to achieve text clustering based on literal similarity.

[0038] In some embodiments, multiple similar texts corresponding to each text are screened, including

[0039] It is determined whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold. If so, each similar text corresponding to each text is added to the candidate text data set corresponding to each text.

[0040] Optionally, a preset text difference threshold is determined based on user input or empirical data.

[0041] It can be understood that by comparing the text difference with a preset text difference threshold and screening multiple similar texts corresponding to each text, texts with high similarity can be screened out, thereby improving the quality of search results and user experience.

[0042] Step 140: Take each text in the candidate text data set as a node, create an edge according to the candidate text data set corresponding to each text, connect the corresponding nodes, and build a graph model.

[0043] Among them, the graph model is a mathematical model used to represent and analyze relational data. The graph model structures the objects (nodes) in the data and the relationships (edges) between nodes into the form of a graph.

[0044] It should be noted that the graph model can express complex text data relationships in a structured way, thereby simplifying the calculation process; the graph model provides an efficient data storage method, especially when processing large-scale text data, it can reduce data storage costs.

[0045] It can be understood that by treating each text in the candidate text dataset as a node and using the candidate text dataset to create edges to build a graph model, the similarity relationship between different texts in the text dataset can be clearly represented and visualized, which helps to accurately find and recommend similar texts and improve the efficiency and accuracy of clustering.

[0046] In some embodiments, each text in the candidate text dataset is used as a node, edges are created according to the candidate text dataset corresponding to each text, and the corresponding nodes are connected to construct a graph model, including: Taking each text in the candidate text dataset as a node, according to the candidate text dataset corresponding to each text, multiple similar text pairs are obtained, and corresponding edges are created to connect the two nodes corresponding to each similar text pair to construct a graph model; According to the text difference of each similar text pair, the weight of the edge corresponding to each similar text pair is determined.

[0047] It can be understood that by treating each text in the candidate text dataset as a node, using similar text pairs to create edges and assigning weights to these edges, it can not only provide more accurate similarity measurement and recommendation results, but also optimize information retrieval, text clustering, classification and other tasks, while revealing the complex relationship between texts and improving computational efficiency.

[0048] Step 150: Using a graph model algorithm to determine multiple maximal connected subgraphs from the graph model, and dividing the node set in each maximal connected subgraph into a cluster.

[0049] Optionally, the graph model algorithm includes a community detection algorithm and a graph traversal algorithm (such as depth-first search and breadth-first search).

[0050] It should be noted that every pair of nodes in a maximal connected subgraph is connected by a path. In a maximal connected subgraph, any two nodes are reachable from each other. A maximal connected subgraph is a connected subgraph that contains as many nodes as possible.

[0051] It can be understood that by finding the maximally connected subgraph, the most densely connected node subset in the graph can be identified, which helps to deeply understand the relationship between nodes in the graph and help to discover the core area of ​​the graph; by dividing the set of nodes in the maximally connected subgraph into a cluster, the accuracy and reliability of clustering can be improved, the clustering effect can be improved, the complexity of calculation can be greatly reduced, and the clustering efficiency can be improved, especially when processing large-scale data.

[0052] In some embodiments, a graph model algorithm is used to determine a plurality of maximal connected subgraphs from a graph model, including: Use graph model algorithm to traverse the graph model; Find multiple maximal connected subgraphs from the graph model.

[0053] Optionally, a depth-first search algorithm or a breadth-first search algorithm is used to traverse the graph model and find all maximal connected subgraphs.

[0054] In some embodiments, after determining a plurality of maximal connected subgraphs from the graph model using a graph model algorithm, the method further includes: The nodes in the graph model that are not included in any maximally connected subgraph are divided into a cluster.

[0055] Figure 2 The second flow chart of the text clustering method provided by the embodiment of the present invention is as follows: Figure 2 As shown, a text clustering method is provided, which includes the following steps: Construct an LSH index based on the text dataset to be clustered; Take each sentence in the data set as a query, search based on the LSH index, and get multiple similar sentences corresponding to each query; Based on each query and multiple similar sentences corresponding to each query, multiple similar sentence pairs are obtained; Take each sentence in the dataset as a node and build a graph based on multiple similar sentence pairs; Traverse the graph and obtain multiple maximal connected subgraphs; Divide the set of nodes in each maximal connected subgraph into a cluster.

[0056] Optionally, multiple similar sentences corresponding to each query are screened to obtain multiple screened similar sentences, and multiple similar sentence pairs are obtained based on each query and the multiple screened similar sentences.

[0057] Optionally, a depth-first search algorithm or a breadth-first search algorithm is used to traverse the graph to obtain multiple maximal connected subgraphs.

[0058] In an embodiment of the present invention, an LSH index of a text data set is constructed, and the LSH index is used to search in the text data set to obtain multiple similar texts corresponding to each text. The multiple similar texts corresponding to each text are screened based on the text difference to obtain a candidate text data set corresponding to each text. Each text in the candidate text data set is used as a node, and an edge is created according to the candidate text data set corresponding to each text. The corresponding nodes are connected to construct a graph model, and a graph model algorithm is used to determine multiple maximal connected subgraphs from the graph model. The node set in each maximal connected subgraph is divided into a cluster cluster. This method can accurately cluster texts with high literal similarity, is suitable for clustering large-scale text data, reduces computing costs, and improves clustering efficiency and real-time performance.

[0059] The text clustering device provided by an embodiment of the present invention is described below. The text clustering device described below and the text clustering method described above can be referred to each other.

[0060] Figure 3 A schematic diagram of the structure of a text clustering device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the text clustering device 300 includes: A first construction unit 310 is used to obtain a text data set to be clustered and construct a local sensitive hash (LSH) index of the text data set; A retrieval unit 320, configured to retrieve texts similar to each text in the text data set by using an LSH index, and obtain multiple similar texts corresponding to each text; A screening unit 330 is used to calculate the text difference between the multiple similar texts corresponding to each text and each text, and screen the multiple similar texts corresponding to each text to obtain a candidate text data set corresponding to each text; The second construction unit 340 takes each text in the candidate text data set as a node, creates an edge according to the candidate text data set corresponding to each text, connects the corresponding nodes, and constructs a graph model; The first clustering unit 350 is used to determine multiple maximal connected subgraphs from the graph model using a graph model algorithm, and divide the node set in each maximal connected subgraph into a cluster.

[0061] Optionally, calculating the text difference between a plurality of similar texts corresponding to each text and each text includes: Calculate the edit distance between multiple similar texts corresponding to each text and each text; Based on the edit distances between the multiple similar texts corresponding to each text and each text, and the length of each text, a normalized text difference between the multiple similar texts corresponding to each text and each text is obtained.

[0062] Optionally, multiple similar texts corresponding to each text are screened, including

[0063] It is determined whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold. If so, each similar text corresponding to each text is added to the candidate text data set corresponding to each text.

[0064] Optionally, a graph model algorithm is used to determine multiple maximal connected subgraphs from the graph model, including: Use graph model algorithm to traverse the graph model; Find multiple maximal connected subgraphs from the graph model.

[0065] Optionally, the text clustering device 300 further includes: The second clustering unit is used to divide the nodes in the graph model that are not added to any maximal connected subgraph into a cluster.

[0066] Optionally, each text in the candidate text dataset is used as a node, edges are created according to the candidate text dataset corresponding to each text, and corresponding nodes are connected to construct a graph model, including: Taking each text in the candidate text dataset as a node, according to the candidate text dataset corresponding to each text, multiple similar text pairs are obtained, and corresponding edges are created to connect the two nodes corresponding to each similar text pair to construct a graph model; According to the text difference of each similar text pair, the weight of the edge corresponding to each similar text pair is determined.

[0067] Optionally, construct a locality sensitive hash (LSH) index for the text dataset, including: Calculate the hash value of each text in the text data set using a hash function; Based on the hash value of each text, generate a corresponding hash table; Build an LSH index based on the hash table.

[0068] It should be noted here that the text clustering device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned text clustering method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0069] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the text clustering method, which includes: obtaining a text data set to be clustered, and constructing a local sensitive hash LSH index of the text data set; using the LSH index, searching the text data set for texts similar to each text in the text data set, and obtaining multiple similar texts corresponding to each text; calculating the text difference between the multiple similar texts corresponding to each text and each text, screening the multiple similar texts corresponding to each text, and obtaining a candidate text data set corresponding to each text; taking each text in the candidate text data set as a node, creating an edge according to the candidate text data set corresponding to each text, connecting the corresponding nodes, and constructing a graph model; using the graph model algorithm to determine multiple maximally connected subgraphs from the graph model, and dividing the node set in each maximally connected subgraph into a cluster cluster.

[0070] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0071] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the text clustering method provided by the above-mentioned methods, the method comprising: obtaining a text data set to be clustered, and constructing a local sensitive hash (LSH) index of the text data set; using the LSH index, searching the text data set for texts similar to each text in the text data set, and obtaining multiple similar texts corresponding to each text; calculating the text difference between the multiple similar texts corresponding to each text and each text, screening the multiple similar texts corresponding to each text, and obtaining a candidate text data set corresponding to each text; taking each text in the candidate text data set as a node, creating edges according to the candidate text data set corresponding to each text, connecting the corresponding nodes, and constructing a graph model; using a graph model algorithm to determine multiple maximal connected subgraphs from the graph model, and dividing the node set in each maximal connected subgraph into a clustering cluster.

[0072] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text clustering method, It is characterized in that include: Obtain a text data set to be clustered, and construct a local sensitive hash (LSH) index of the text data set; Using the LSH index, searching the text data set for texts similar to each text in the text data set to obtain a plurality of similar texts corresponding to each text; Calculating the text difference between a plurality of similar texts corresponding to each text and each text, screening the plurality of similar texts corresponding to each text, and obtaining a candidate text data set corresponding to each text; Taking each text in the candidate text data set as a node, creating an edge according to the candidate text data set corresponding to each text, connecting the corresponding nodes, and constructing a graph model; A graph model algorithm is used to determine a plurality of maximal connected subgraphs from the graph model, and a set of nodes in each maximal connected subgraph is divided into a cluster.

2. The text clustering method according to claim 1, It is characterized in that The calculating the text difference between the multiple similar texts corresponding to each text and each text includes: Calculating the edit distance between a plurality of similar texts corresponding to each text and each text; Based on the edit distances between the multiple similar texts corresponding to each text and each text, and the length of each text, a normalized text difference between the multiple similar texts corresponding to each text and each text is obtained.

3. The text clustering method according to claim 1, It is characterized in that The screening of the multiple similar texts corresponding to each text includes: It is determined whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold; if so, each similar text corresponding to each text is added to the candidate text data set corresponding to each text.

4. The text clustering method according to claim 1, It is characterized in that A graph model algorithm is used to determine a plurality of maximal connected subgraphs from the graph model, including: Using a graph model algorithm to traverse the graph model; Find a plurality of maximal connected subgraphs from the graph model.

5. The text clustering method according to claim 1, It is characterized in that After the graph model algorithm is used to determine a plurality of maximal connected subgraphs from the graph model, the method further includes: The nodes in the graph model that are not added to any maximally connected subgraph are divided into a cluster.

6. The text clustering method according to claim 1, It is characterized in that The method takes each text in the candidate text data set as a node, creates an edge according to the candidate text data set corresponding to each text, connects the corresponding nodes, and constructs a graph model, including: Taking each text in the candidate text data set as a node, obtaining a plurality of similar text pairs according to the candidate text data set corresponding to each text, creating corresponding edges to connect two nodes corresponding to each similar text pair, and constructing a graph model; The weight of the edge corresponding to each similar text pair is determined according to the text difference degree of each similar text pair.

7. The text clustering method according to any one of claims 2 to 6, It is characterized in that The step of constructing a local sensitive hash (LSH) index of the text data set includes: Calculate the hash value of each text in the text data set by using a hash function; Based on the hash value of each text, generate a corresponding hash table; Based on the hash table, the LSH index is constructed.

8. A text clustering device, It is characterized in that include: A first construction unit is used to obtain a text data set to be clustered and construct a local sensitive hash (LSH) index of the text data set; A retrieval unit, used to retrieve texts similar to each text in the text data set in the text data set by using the LSH index, and obtain multiple similar texts corresponding to each text; A screening unit, used for calculating the text difference between a plurality of similar texts corresponding to each text and each text, screening the plurality of similar texts corresponding to each text, and obtaining a candidate text data set corresponding to each text; A second construction unit, taking each text in the candidate text data set as a node, creating an edge according to the candidate text data set corresponding to each text, connecting the corresponding nodes, and constructing a graph model; The first clustering unit is used to determine a plurality of maximal connected subgraphs from the graph model by using a graph model algorithm, and divide the node set in each maximal connected subgraph into a cluster.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the text clustering method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the text clustering method according to any one of claims 1 to 7 is implemented.

Images