Text clustering method and apparatus, electronic device, and storage medium

Abstract

The application provides a text clustering method and device, electronic equipment and storage medium, wherein the method comprises: obtaining a text data set to be clustered, and constructing an LSH index of the text data set; retrieving in the text data set by using the LSH index to obtain a plurality of similar texts corresponding to each text; calculating a text difference degree, screening the plurality of similar texts corresponding to each text to obtain a candidate text data set corresponding to each text; taking each text in the candidate text data set as a node, creating an edge according to the candidate text data set corresponding to each text, connecting the corresponding nodes, and constructing a graph model; and determining a plurality of maximal connected subgraphs from the graph model by using a graph model algorithm, and dividing a node set in each maximal connected subgraph into a clustering cluster. The application can accurately cluster texts with high literal similarity, is suitable for clustering large-scale text data, reduces the calculation cost, and improves the clustering efficiency and real-time performance.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a text clustering method, apparatus, electronic device, and storage medium. Background Technology

[0002] Text clustering is an important task in the field of natural language processing. Its main purpose is to group similar texts into the same category to facilitate information organization, management, and retrieval. Moreover, in some specific domains, text clustering is required based on the literal similarity of the texts.

[0003] Existing text clustering methods mainly convert text into vector representations and then combine them with methods such as K-means and hierarchical clustering. However, while vector representations are effective in capturing semantic similarity, they are not accurate enough in judging literal similarity. Traditional clustering methods are computationally expensive and inefficient when processing large-scale text data, making it difficult to meet real-time requirements. Summary of the Invention

[0004] This invention provides a text clustering method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that use text to convert it into vector representations and then combine it with methods such as K-means and hierarchical clustering. These methods are not accurate enough in judging literal similarity, and when processing large-scale text data, they are computationally expensive, inefficient, and fail to meet real-time requirements.

[0005] In a first aspect, the present invention provides a text clustering method, comprising: Obtain the text dataset to be clustered, and construct a Locality Sensitive Hash (LSH) index for the text dataset; Using the LSH index, texts similar to each text in the text dataset are retrieved, resulting in multiple similar texts corresponding to each text. Calculate the text difference between each text and multiple similar texts corresponding to each text, filter the multiple similar texts corresponding to each text, and obtain the candidate text dataset corresponding to each text; Using each text in the candidate text dataset as a node, create edges based on the candidate text dataset corresponding to each text, connect the corresponding nodes, and construct a graph model; A graph model algorithm is used to determine multiple maximal connected subgraphs from the graph model, and the set of nodes in each maximal connected subgraph is divided into a cluster.

[0006] In some embodiments, calculating the text difference between each text and a plurality of similar texts corresponding to each text includes: Calculate the edit distance between each text and multiple similar texts corresponding to each text; Based on the edit distance between each text and its multiple similar texts, and the length of each text, the normalized text difference between each text and its multiple similar texts is obtained.

[0007] In some embodiments, the filtering of multiple similar texts corresponding to each text includes...

[0008] Determine whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold. If so, add each similar text corresponding to each text to the candidate text dataset corresponding to each text.

[0009] In some embodiments, a graph model algorithm is used to determine multiple maximal connected subgraphs from the graph model, including: The graph model is traversed using a graph model algorithm; Identify multiple maximal connected subgraphs from the graph model.

[0010] In some embodiments, after determining multiple maximally connected subgraphs from the graph model using a graph model algorithm, the method further includes: The nodes in the graph model that are not included in any maximal connected subgraph are divided into a cluster.

[0011] In some embodiments, the step of using each text in the candidate text dataset as a node, creating edges based on the candidate text dataset corresponding to each text, connecting the corresponding nodes, and constructing a graph model includes: Using each text in the candidate text dataset as a node, and based on the candidate text dataset corresponding to each text, multiple similar text pairs are obtained. Corresponding edges are created to connect the two nodes corresponding to each similar text pair, and a graph model is constructed. The weight of the edge corresponding to each similar text pair is determined based on the text difference degree of each similar text pair.

[0012] In some embodiments, constructing the Locality Sensitive Hash (LSH) index of the text dataset includes: Calculate the hash value of each text in the text dataset using a hash function; Based on the hash value of each text, a corresponding hash table is generated; Based on the hash table, construct the LSH index.

[0013] Secondly, the present invention also provides a text clustering apparatus, comprising: The first building unit is used to obtain the text dataset to be clustered and to build a Locality Sensitive Hash (LSH) index of the text dataset. The retrieval unit is used to retrieve texts similar to each text in the text dataset using the LSH index, and to obtain multiple similar texts corresponding to each text. The filtering unit is used to calculate the text difference between each text and multiple similar texts corresponding to each text, and to filter the multiple similar texts corresponding to each text to obtain a candidate text dataset corresponding to each text. The second construction unit uses each text in the candidate text dataset as a node, creates edges based on the candidate text dataset corresponding to each text, connects the corresponding nodes, and constructs a graph model. The first clustering unit is used to determine multiple maximally connected subgraphs from the graph model using a graph model algorithm, and to divide the set of nodes in each maximally connected subgraph into a cluster.

[0014] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the text clustering methods described above.

[0015] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text clustering method described above.

[0016] The text clustering method, apparatus, electronic device, and storage medium provided by this invention construct an LSH index for the text dataset, use the LSH index to retrieve multiple similar texts corresponding to each text in the text dataset, filter the multiple similar texts corresponding to each text based on text differences to obtain a candidate text dataset corresponding to each text, use each text in the candidate text dataset as a node, create edges according to the candidate text dataset corresponding to each text, connect the corresponding nodes to construct a graph model, use a graph model algorithm to determine multiple maximally connected subgraphs from the graph model, and divide the set of nodes in each maximally connected subgraph into a cluster. It can accurately cluster texts with high literal similarity, is suitable for clustering large-scale text data, reduces computational costs, and improves the efficiency and real-time performance of clustering. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the text clustering method provided in this embodiment of the invention; Figure 2 This is the second flowchart illustrating the text clustering method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the structure of the text clustering device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Figure 1 This is one of the flowcharts illustrating the text clustering method provided in an embodiment of the present invention. For example... Figure 1 As shown, a text clustering method is provided, including the following steps: step 110, step 120, step 130, step 140, and step 150. This method's steps are merely one possible implementation of the present invention.

[0021] Step 110: Obtain the text dataset to be clustered and construct the Local Sensitive Hash (LSH) index of the text dataset.

[0022] Locality-Sensitive Hashing (LSH) is an approximate nearest neighbor search technique for high-dimensional data, suitable for similarity searches in text data. The core idea of ​​LSH is to map similar data points into the same bucket using a hash function, thereby reducing the number of candidate pairs that need to be compared and improving search efficiency.

[0023] In some embodiments, constructing a Locality Sensitive Hash (LSH) index for the text dataset includes: Calculate the hash value of each text in the text dataset using a hash function; Generate a corresponding hash table based on the hash value of each text. Based on the hash table, construct the LSH index.

[0024] Optionally, the text dataset can be preprocessed to convert the text data into a form suitable for processing by a hash function.

[0025] Optionally, a suitable hash function can be selected based on the type of text data and the method of similarity measurement.

[0026] Optionally, the parameters of the hash function can be optimized and adjusted.

[0027] Step 120: Using the LSH index, retrieve texts similar to each text in the text dataset to obtain multiple similar texts corresponding to each text.

[0028] Optionally, each text in the text dataset can be treated as a query, and based on each query, a retrieval can be performed in the text dataset to obtain multiple similar texts corresponding to each text.

[0029] For example, for a query, the LSH index is used to retrieve texts similar to the query in the text dataset, resulting in three similar texts: d1, d2, and d3.

[0030] Understandably, by utilizing LSH indexes, texts similar to each text in a text dataset can be retrieved, reducing computational and storage costs, improving retrieval efficiency and real-time performance, and making it suitable for large-scale datasets.

[0031] Step 130: Calculate the text difference between each text and its multiple similar texts, and filter the multiple similar texts corresponding to each text to obtain the candidate text dataset corresponding to each text.

[0032] Understandably, by calculating the text difference, multiple similar texts corresponding to each text can be filtered based on the text difference, thereby obtaining a candidate text dataset with high similarity to each text.

[0033] In some embodiments, calculating the text difference between each text and multiple similar texts corresponding to each text includes: Calculate the edit distance between each text and its multiple similar texts; Based on the edit distance between each text and its multiple similar texts, and the length of each text, the normalized text difference between each text and its multiple similar texts is obtained.

[0034] Edit distance refers to the minimum number of single-character edit operations required to transform one string into another; these edit operations include insertion, deletion, and replacement. Edit distance is used to measure the difference between two strings.

[0035] For example, if the similar texts corresponding to the query are d1, d2, and d3, then the edit distances of (query, d1), (query, d2), and (query, d3) are calculated respectively.

[0036] Optionally, the edit distance can be divided by the query length to obtain the normalized text difference.

[0037] Understandably, calculating text dissimilarity through edit distance can help achieve text clustering based on literal similarity.

[0038] In some embodiments, multiple similar texts corresponding to each text are filtered, including...

[0039] Determine whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold. If so, add each similar text corresponding to each text to the candidate text dataset corresponding to each text.

[0040] Optionally, a preset text difference threshold can be determined based on user input or experience data.

[0041] Understandably, by comparing the text difference with a preset text difference threshold, multiple similar texts corresponding to each text can be filtered out, thus improving the quality of search results and enhancing user experience.

[0042] Step 140: Using each text in the candidate text dataset as a node, create edges based on the candidate text dataset corresponding to each text, connect the corresponding nodes, and construct a graph model.

[0043] Among them, the graph model is a mathematical model used to represent and analyze relational data. The graph model structures the objects (nodes) in the data and the relationships (edges) between nodes into the form of a graph.

[0044] It should be noted that graph models can express complex textual data relationships in a structured way, thereby simplifying the calculation process; graph models provide an efficient way of data storage, especially when processing large-scale textual data, and can reduce data storage costs.

[0045] Understandably, by using each text in the candidate text dataset as a node and creating edges using the candidate text dataset to construct a graph model, the similarity relationships between different texts in the text dataset can be clearly represented and visualized, which helps to accurately identify and recommend similar texts and improve the efficiency and accuracy of clustering.

[0046] In some embodiments, using each text in the candidate text dataset as a node, edges are created based on the candidate text dataset corresponding to each text, connecting the corresponding nodes to construct a graph model, including: Using each text in the candidate text dataset as a node, based on the candidate text dataset corresponding to each text, multiple similar text pairs are obtained, and corresponding edges are created to connect the two nodes corresponding to each similar text pair to construct a graph model. The weight of the edge corresponding to each similar text pair is determined based on the text difference degree of each similar text pair.

[0047] Understandably, by treating each text in the candidate text dataset as a node and creating edges using similar text pairs and assigning weights to these edges, we can not only provide more accurate similarity measurements and recommendation results, but also optimize tasks such as information retrieval, text clustering, and classification, while revealing complex relationships between texts and improving computational efficiency.

[0048] Step 150: Use a graph model algorithm to determine multiple maximal connected subgraphs from the graph model, and divide the set of nodes in each maximal connected subgraph into a cluster.

[0049] Optionally, graph model algorithms include community detection algorithms and graph traversal algorithms (such as depth-first search and breadth-first search).

[0050] It should be noted that every pair of nodes in a maximal connected subgraph is connected by a path. In a maximal connected subgraph, any two nodes are reachable from each other. A maximal connected subgraph is a connected subgraph that contains as many nodes as possible.

[0051] Understandably, by finding a maximally connected subgraph, we can identify the most tightly connected subset of nodes in the graph, which helps to understand the relationships between nodes in the graph and helps to discover the core region of the graph. By dividing the set of nodes in the maximally connected subgraph into a cluster, we can improve the accuracy and reliability of clustering, improve the clustering effect, significantly reduce the computational complexity, and improve the clustering efficiency, especially when dealing with large-scale data.

[0052] In some embodiments, a graph model algorithm is used to determine multiple maximal connected subgraphs from a graph model, including: The graph model is traversed using a graph model algorithm; Find multiple maximal connected subgraphs from the graph model.

[0053] Optionally, a depth-first search algorithm or a breadth-first search algorithm can be used to traverse the graph model and find all maximal connected subgraphs.

[0054] In some embodiments, after determining multiple maximal connected subgraphs from a graph model using a graph model algorithm, the method further includes: The nodes in the graph model that are not included in any maximal connected subgraph are grouped into a cluster.

[0055] Figure 2 This is a second flowchart illustrating the text clustering method provided in this embodiment of the invention, as shown below. Figure 2 As shown, a text clustering method is provided, which includes the following steps: Based on the text dataset to be clustered, construct an LSH index; Each sentence in the dataset is treated as a query, and a search is performed based on the LSH index to obtain multiple similar sentences corresponding to each query. Based on each query and the multiple similar sentences corresponding to each query, multiple pairs of similar sentences are obtained; Using each sentence in the dataset as a node, construct a graph based on multiple similar sentence pairs; Traversing the graph yields multiple maximal connected subgraphs; The set of nodes in each maximally connected subgraph is divided into a cluster.

[0056] Optionally, multiple similar sentences corresponding to each query can be filtered to obtain multiple filtered similar sentences, and multiple similar sentence pairs can be obtained based on each query and the multiple filtered similar sentences.

[0057] Optionally, a depth-first search algorithm or a breadth-first search algorithm can be used to traverse the graph and obtain multiple maximal connected subgraphs.

[0058] In this embodiment of the invention, an LSH index is constructed for the text dataset. Using the LSH index, multiple similar texts corresponding to each text are retrieved from the text dataset. Based on the text difference, the multiple similar texts corresponding to each text are filtered to obtain a candidate text dataset corresponding to each text. Each text in the candidate text dataset is used as a node. Edges are created according to the candidate text dataset corresponding to each text to connect the corresponding nodes, thus constructing a graph model. A graph model algorithm is used to determine multiple maximally connected subgraphs from the graph model. The set of nodes in each maximally connected subgraph is divided into a cluster. This method can accurately cluster texts with high literal similarity, is suitable for clustering large-scale text data, reduces computational costs, and improves the efficiency and real-time performance of clustering.

[0059] The text clustering device provided in the embodiments of the present invention will be described below. The text clustering device described below can be referred to in correspondence with the text clustering method described above.

[0060] Figure 3 This is a schematic diagram of the structure of the text clustering device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the text clustering device 300 includes: The first building unit 310 is used to obtain the text dataset to be clustered and build the Local Sensitive Hash (LSH) index of the text dataset; The retrieval unit 320 is used to retrieve texts similar to each text in the text dataset using the LSH index, and obtain multiple similar texts corresponding to each text. The filtering unit 330 is used to calculate the text difference between each text and multiple similar texts corresponding to each text, and to filter the multiple similar texts corresponding to each text to obtain the candidate text dataset corresponding to each text. The second construction unit 340 uses each text in the candidate text dataset as a node, creates an edge based on the candidate text dataset corresponding to each text, connects the corresponding nodes, and constructs a graph model. The first clustering unit 350 is used to determine multiple maximally connected subgraphs from the graph model using graph model algorithms, and to divide the set of nodes in each maximally connected subgraph into a cluster.

[0061] Optionally, the text difference between each text and its multiple similar texts is calculated, including: Calculate the edit distance between each text and its multiple similar texts; Based on the edit distance between each text and its multiple similar texts, and the length of each text, the normalized text difference between each text and its multiple similar texts is obtained.

[0062] Optionally, multiple similar texts corresponding to each text can be filtered, including...

[0063] Determine whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold. If so, add each similar text corresponding to each text to the candidate text dataset corresponding to each text.

[0064] Optionally, a graph model algorithm is used to determine multiple maximal connected subgraphs from the graph model, including: The graph model is traversed using a graph model algorithm; Find multiple maximal connected subgraphs from the graph model.

[0065] Optionally, the text clustering device 300 also includes: The second clustering unit is used to divide the nodes in the graph model that have not been added to any maximal connected subgraph into a cluster.

[0066] Optionally, using each text in the candidate text dataset as a node, edges are created based on the candidate text dataset corresponding to each text, connecting the corresponding nodes to construct a graph model, including: Using each text in the candidate text dataset as a node, based on the candidate text dataset corresponding to each text, multiple similar text pairs are obtained, and corresponding edges are created to connect the two nodes corresponding to each similar text pair to construct a graph model. The weight of the edge corresponding to each similar text pair is determined based on the text difference degree of each similar text pair.

[0067] Optionally, a Locality Sensitive Hash (LSH) index is constructed for the text dataset, including: Calculate the hash value of each text in the text dataset using a hash function; Generate a corresponding hash table based on the hash value of each text. Based on the hash table, construct the LSH index.

[0068] It should be noted that the text clustering device provided in this embodiment of the invention can implement all the method steps implemented in the above-described text clustering method embodiments and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail here.

[0069] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a text clustering method, which includes: acquiring a text dataset to be clustered and constructing a Local Sensitive Hash (LSH) index for the text dataset; using the LSH index, retrieving texts similar to each text in the text dataset to obtain multiple similar texts corresponding to each text; calculating the text difference between the multiple similar texts corresponding to each text and each text, filtering the multiple similar texts corresponding to each text to obtain a candidate text dataset corresponding to each text; using each text in the candidate text dataset as a node, creating edges according to the candidate text dataset corresponding to each text, connecting the corresponding nodes, and constructing a graph model; using a graph model algorithm to determine multiple maximally connected subgraphs from the graph model, and dividing the set of nodes in each maximally connected subgraph into a cluster.

[0070] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the text clustering method provided by the above methods. The method includes: acquiring a text dataset to be clustered and constructing a Local Sensitive Hash (LSH) index for the text dataset; using the LSH index, retrieving texts similar to each text in the text dataset to obtain multiple similar texts corresponding to each text; calculating the text difference between the multiple similar texts corresponding to each text and each text, filtering the multiple similar texts corresponding to each text to obtain a candidate text dataset corresponding to each text; using each text in the candidate text dataset as a node, creating edges according to the candidate text dataset corresponding to each text, connecting the corresponding nodes, and constructing a graph model; using a graph model algorithm to determine multiple maximally connected subgraphs from the graph model, and dividing the set of nodes in each maximally connected subgraph into a cluster.

[0072] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text clustering method, characterized in that, include: Obtain the text dataset to be clustered, select a hash function based on the type of each text in the text dataset and the similarity measurement method, optimize and adjust the parameters of the hash function, and construct a Local Sensitive Hash (LSH) index for the text dataset; Using the LSH index, texts similar to each text in the text dataset are retrieved, resulting in multiple similar texts corresponding to each text. Calculate the text difference between each text and multiple similar texts corresponding to each text, filter the multiple similar texts corresponding to each text, and obtain the candidate text dataset corresponding to each text; Using each text in the candidate text dataset as a node, create edges based on the candidate text dataset corresponding to each text, connect the corresponding nodes, and construct a graph model; A graph model algorithm is used to determine multiple maximally connected subgraphs from the graph model, and the set of nodes in each maximally connected subgraph is divided into a cluster. The filtering of multiple similar texts corresponding to each text includes: Determine whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold. If so, add each similar text corresponding to each text to the candidate text dataset corresponding to each text. The step of constructing a graph model by using each text in the candidate text dataset as a node, creating edges based on the candidate text dataset corresponding to each text, connecting the corresponding nodes, and then proceeding as follows: Using each text in the candidate text dataset as a node, and based on the candidate text dataset corresponding to each text, multiple similar text pairs are obtained. Corresponding edges are created to connect the two nodes corresponding to each similar text pair, and a graph model is constructed. The weight of the edge corresponding to each similar text pair is determined based on the text difference degree of each similar text pair; The calculation of the text difference between each text and multiple similar texts includes: Calculate the edit distance between each text and multiple similar texts corresponding to each text; Based on the edit distance between each text and its multiple similar texts, and the length of each text, the normalized text difference between each text and its multiple similar texts is obtained.

2. The text clustering method according to claim 1, characterized in that, A graph model algorithm is used to determine multiple maximal connected subgraphs from the graph model, including: The graph model is traversed using a graph model algorithm; Identify multiple maximal connected subgraphs from the graph model.

3. The text clustering method according to claim 1, characterized in that, After determining multiple maximal connected subgraphs from the graph model using a graph model algorithm, the method further includes: The nodes in the graph model that are not included in any maximal connected subgraph are divided into a cluster.

4. The text clustering method according to any one of claims 1-3, characterized in that, The construction of the Locality Sensitive Hash (LSH) index for the text dataset includes: Calculate the hash value of each text in the text dataset using a hash function; Based on the hash value of each text, a corresponding hash table is generated; Based on the hash table, construct the LSH index.

5. A text clustering device, characterized in that, include: The first construction unit is used to obtain the text dataset to be clustered, select a hash function according to the type of each text in the text dataset and the similarity measurement method, optimize and adjust the parameters of the hash function, and construct the Local Sensitive Hash (LSH) index of the text dataset. The retrieval unit is used to retrieve texts similar to each text in the text dataset using the LSH index, and to obtain multiple similar texts corresponding to each text. The filtering unit is used to calculate the text difference between each text and multiple similar texts corresponding to each text, and to filter the multiple similar texts corresponding to each text to obtain a candidate text dataset corresponding to each text. The second construction unit is used to construct a graph model by using each text in the candidate text dataset as a node, creating edges according to the candidate text dataset corresponding to each text, connecting the corresponding nodes, and constructing a graph model. The first clustering unit is used to determine multiple maximally connected subgraphs from the graph model using a graph model algorithm, and to divide the set of nodes in each maximally connected subgraph into a cluster. The filtering of multiple similar texts corresponding to each text includes: Determine whether the text difference between each similar text corresponding to each text and each text is less than a preset text difference threshold. If so, add each similar text corresponding to each text to the candidate text dataset corresponding to each text. The step of constructing a graph model by using each text in the candidate text dataset as a node, creating edges based on the candidate text dataset corresponding to each text, connecting the corresponding nodes, and then proceeding as follows: Using each text in the candidate text dataset as a node, and based on the candidate text dataset corresponding to each text, multiple similar text pairs are obtained. Corresponding edges are created to connect the two nodes corresponding to each similar text pair, and a graph model is constructed. The weight of the edge corresponding to each similar text pair is determined based on the text difference degree of each similar text pair; The calculation of the text difference between each text and multiple similar texts includes: Calculate the edit distance between each text and multiple similar texts corresponding to each text; Based on the edit distance between each text and its multiple similar texts, and the length of each text, the normalized text difference between each text and its multiple similar texts is obtained.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the text clustering method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text clustering method as described in any one of claims 1 to 4.

Images