Text clustering method, apparatus, and readable storage medium

Abstract

Embodiments of the present application disclose a text clustering method, device and readable storage medium, the text clustering method comprising: obtaining at least one text corpus; performing vectorization processing on each text corpus in the at least one text corpus to obtain at least one text vector corresponding to the at least one text corpus; performing similarity calculation on the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector; splitting the text similarity matrix to obtain K text similarity blocks, and obtaining N text clusters according to the K text similarity blocks, each text similarity block in the K text similarity blocks including a similarity greater than or equal to a similarity threshold value, and the number of the similarity being greater than or equal to a splitting number threshold value, wherein K is an integer greater than or equal to 1, and N is a positive integer less than or equal to K. By using such a text clustering method, the clustering speed and the accuracy of the clustering result can be improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing applications, and more particularly to a text clustering method, apparatus, server, and readable storage medium. Background Technology

[0002] Intelligent customer service robots primarily function by providing intelligent question-and-answer support and guiding customers through self-service processes. They can typically resolve most customer issues quickly, improving customer service efficiency and customer satisfaction. Text, as one of the main carriers of information data, plays a crucial role in intelligent customer service robot systems.

[0003] In order for intelligent customer service robots to match the best answers to customer questions, it is necessary to cluster a large number of customer questions into text information. However, due to the high space complexity of text clustering algorithms and the uneven inter-class spacing of text information data, text clustering is slow, inefficient, and inaccurate.

[0004] It is evident that improving the speed and accuracy of clustering results in text clustering tasks is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a text clustering method, apparatus, server, and readable storage medium. Using this text clustering method can improve the clustering speed and accuracy of the clustering results when performing text clustering tasks.

[0006] In a first aspect, embodiments of the present invention provide a text clustering method, the method comprising:

[0007] Obtain at least one text corpus;

[0008] Each text corpus in the at least one text corpus is vectorized to obtain at least one text vector corresponding to the at least one text corpus;

[0009] The similarity of the at least one text vector is calculated to obtain the text similarity matrix corresponding to the at least one text vector;

[0010] The text similarity matrix is ​​segmented to obtain K text similarity blocks, and N text clusters are obtained based on the K text similarity blocks. The similarity of each text similarity block in the K text similarity blocks is greater than or equal to a similarity threshold, and the number of similarities is greater than or equal to a segmentation number threshold. K is an integer greater than or equal to 1, and N is a positive integer less than or equal to K.

[0011] Secondly, embodiments of the present invention provide a text clustering apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire at least one text corpus.

[0013] The processing module is used to perform vectorization processing on each text corpus in the at least one text corpus to obtain at least one text vector corresponding to the at least one text corpus;

[0014] The processing module is further configured to perform similarity calculation on the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector;

[0015] The processing module is further configured to segment the text similarity matrix to obtain K text similarity blocks, and obtain N text clusters based on the K text similarity blocks. The similarity of each text similarity block in the K text similarity blocks is greater than or equal to a similarity threshold, and the number of similarities is greater than or equal to a segmentation number threshold. K is an integer greater than or equal to 1, and N is a positive integer less than or equal to K.

[0016] Thirdly, embodiments of the present invention provide a server, the server including an input device and an output device, the server further including a processor adapted to implement one or more instructions, the one or more instructions being adapted to be loaded by the processor and executed by the text clustering method described in the first aspect above.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the text clustering method described in the first aspect.

[0018] In this embodiment, the server acquires at least one text corpus and vectorizes each text corpus to obtain at least one text vector corresponding to the at least one text corpus. Further, the server can calculate the similarity of the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector, and segment the text similarity matrix to obtain K text similarity blocks. Then, based on these K text similarity blocks, N text clusters can be obtained, wherein the similarity of each text similarity block in the K text similarity blocks is greater than or equal to a similarity threshold, and the number of similarities is greater than or equal to a segmentation number threshold, where K is an integer greater than or equal to 1, and N is a positive integer less than or equal to K. Using such a text clustering method can improve the clustering speed and accuracy of the clustering results when performing text clustering tasks. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a text clustering method provided in an embodiment of the present invention;

[0021] Figure 2 This is a flowchart illustrating another text clustering method provided in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of a text cluster provided in an embodiment of the present invention;

[0023] Figure 4 This is a flowchart illustrating another text clustering method provided in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the structure of a text clustering device provided in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Text clustering algorithms suffer from high space complexity and uneven inter-class spacing in text data, resulting in slow clustering speed, low efficiency, and low accuracy. This application addresses this issue by calculating the similarity of text vectors within the text corpus to obtain a text similarity matrix. Further, this similarity matrix is ​​segmented to obtain at least one text similarity block, and at least one text cluster is derived based on this block. This method, by identifying similar text clusters from each segmented similarity block, improves both the speed and accuracy of text clustering tasks.

[0028] The text clustering method mentioned in the technical solution of this application can be applied to a server, including but not limited to an application server or a server cluster.

[0029] Please see Figure 1 This is a flowchart illustrating a text clustering method provided in an embodiment of the present invention. The text clustering method includes the following steps:

[0030] S101: Obtain at least one text corpus.

[0031] The server receives at least one or more text corpora input from an application on a user's terminal. For example, if an e-commerce application A is installed on the user's terminal, and at least one or more users input at least one chat question (i.e., the aforementioned text corpora) through e-commerce application A, the server obtains (or receives) the at least one chat question.

[0032] S102: Vectorize each text corpus in at least one text corpus to obtain at least one text vector corresponding to at least one text corpus.

[0033] It is important to know that vectorization is a process that converts each text corpus into a text vector.

[0034] The server performs vectorization processing on each text corpus obtained above to obtain at least one text vector, wherein the number of text vectors is the same as the number of text corpora, and the text vectors correspond one-to-one with the text corpora.

[0035] In one embodiment, the server acquires at least one text corpus, segments each text corpus into it to obtain individual words, and performs part-of-speech tagging on each word. Further, the server can calculate the inverse document frequency (INF) value of each word in each text corpus, and vectorize each text corpus based on the INF tagging results and the INF values ​​of each word in each text corpus, obtaining at least one text vector corresponding to at least one text corpus.

[0036] Among them, the specific way for the server to vectorize each text corpus according to the词性标注结果 (POS tagging result) and the inverse document frequency value of each word segment in each text corpus can be to correct the inverse document frequency value of each word segment according to the词性 of each word segment. That is, obtain the preset weight value corresponding to the词性 of each word segment, and respectively use the preset weight value corresponding to the词性 of each word segment to perform weighted calculation on the inverse document frequency value of each word segment in each text corpus, so as to obtain the weighted inverse document frequency value of each word segment. Further, the weighted inverse document frequency value of each word segment can be used to represent each text corpus, and then the text vector corresponding to each text corpus can be obtained. Among them, the preset weight value corresponding to the词性 of each word segment is determined by developers according to the experimental scenario data and can be adjusted accordingly according to the specific application scenario later, and no specific limitation is made here.

[0037] Exemplarily, the server can use the jieba word segmentation technology to perform word segmentation processing and词性标注 (POS tagging) on each text corpus. For example, using the jieba word segmentation technology to segment the text corpus "我穿什么码" into the noun "我", the verb "穿", the pronoun "什么", and the measure word "码", and according to the following formula:

[0038]

[0039] Calculate the inverse document frequency of each word segment in each text corpus, and obtain that the inverse document frequency value of "我" is 0.6, the inverse document frequency value of "穿" is 2, the inverse document frequency value of "什么" is 1.5, and the inverse document frequency value of "码" is 1. Obtain the weight value corresponding to each word segment. The weight value corresponding to the noun "我" is 0.5, the weight value corresponding to the verb "穿" is 1.2, the weight value corresponding to the pronoun "什么" is 1.2, and the weight corresponding to the measure word "码" is 1.2. Then, perform weighted calculation on the inverse document frequency value corresponding to each word segment according to the词性 of each word segment, and obtain that the weighted inverse document frequency value of the noun "我" is 0.3, the weighted inverse document frequency value of the verb "穿" is 2.4, the weighted inverse document frequency value of the pronoun "什么" is 1.8, and the weighted inverse document frequency value of the measure word "码" is 1.2. Then, the text vector corresponding to "我穿什么码" can be expressed as [0.3, 0.4, 1.8, 1.2].

[0040] Optionally, the server can also vectorize each text corpus according to the bag-of-words model algorithm. The basic principle of the bag-of-words model algorithm is a prior art means and will not be elaborated here too much.

[0041] S103: Calculate the similarity of at least one text vector to obtain the text similarity matrix corresponding to the at least one text vector.

[0042] The server calculates the similarity between each text vector in at least one text vector and other text vectors, and constructs a text similarity matrix based on the calculated similarity.

[0043] In one embodiment, the server can calculate the similarity between any two text vectors in at least one text vector according to a distributed computing method, and construct a text similarity matrix corresponding to at least one text vector based on the similarity between the vectors.

[0044] For example, the server acquires multiple short text corpora. The server indexes these short texts from 0 to the total number of short texts, and then pairs all the short texts together to obtain the total similarity calculation task. Based on the number of application servers, this total similarity calculation task is distributed, with each application server calculating a portion of the total similarity calculation task. This method improves computation speed. Specifically, the server can calculate the cosine similarity (the cosine of the angle between two text vectors) to measure the similarity between any two text vectors. Finally, the server aggregates the similarity calculation results from each application server according to their index numbers to obtain a text similarity matrix with similarity as the element. As an optional implementation, the server can also measure the similarity between any two text vectors using methods such as Euclidean distance, Manhattan distance, or Pearson correlation coefficient.

[0045] S104: Segment the text similarity matrix to obtain K text similarity blocks, and obtain N text clusters based on the K text similarity blocks. In the K text similarity blocks, the similarity of each text similarity block is greater than or equal to the similarity threshold, and the number of similarities is greater than or equal to the segmentation number threshold. K is an integer greater than or equal to 1, and N is a positive integer less than or equal to K.

[0046] The similarity threshold was calculated by the developers based on experimental data and can be adjusted according to specific application scenarios; no specific restrictions are imposed here. The segmentation quantity threshold can also be calculated by the developers based on experimental data and can be adjusted according to specific application scenarios; no specific restrictions are imposed here. Alternatively, the segmentation quantity threshold can also be dynamically adjusted according to specific application scenarios, and its calculation formula is as follows:

[0047]

[0048] Where simNum is the number of elements in the text similarity matrix that are greater than the similarity threshold, and n is the total number of texts.

[0049] The server can remove elements in the text similarity matrix that are less than the similarity threshold to obtain an effective text similarity matrix. It can then segment the effective text similarity matrix according to the segmentation quantity threshold to obtain K similarity blocks, and obtain N text clusters based on the K similarity blocks, where K is an integer greater than or equal to 1 and N is a positive integer less than or equal to K.

[0050] In one embodiment, the server scans the text similarity matrix row by row to obtain at least one row vector included in the text similarity matrix, and counts the number of valid elements in each row vector whose corresponding similarity is greater than or equal to a similarity threshold. Further, the server can take the row vectors whose number of corresponding valid elements in at least one row vector is greater than or equal to a segmentation number threshold as text similarity blocks to obtain K text similarity blocks.

[0051] Optionally, the server can also scan the text similarity matrix column by column to obtain at least one column vector included in the text similarity matrix, and count the number of valid elements whose similarity is greater than or equal to the similarity threshold in each column vector. Further, the server can use the column vectors whose number of valid elements in at least one column vector is greater than or equal to the segmentation number threshold as text similarity blocks to obtain K text similarity blocks.

[0052] For example, with a similarity threshold of 0.4, the text similarity matrix A corresponds to row vectors a1, a2, and a3, each with 1000 elements. After the server removes elements with similarity less than 0.4 from row vectors a1, a2, and a3, the number of elements in row vector a1 becomes 300, a2 becomes 100, and a3 becomes 80. Assuming the segmentation threshold calculated using the aforementioned formula is 20, since the number of elements in row vectors a1, a2, and a3 are all greater than 20, they are treated as three text similarity blocks.

[0053] In this embodiment, the server acquires at least one text corpus and vectorizes each text corpus to obtain at least one text vector corresponding to the at least one text corpus. Further, the server can calculate the similarity of the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector, and segment the text similarity matrix to obtain K text similarity blocks. Then, based on these K text similarity blocks, N text clusters can be obtained, wherein the similarity of each text similarity block in the K text similarity blocks is greater than or equal to a similarity threshold, and the number of similarities is greater than or equal to a segmentation number threshold, where K is an integer greater than or equal to 1, and N is a positive integer less than or equal to K. Using such a text clustering method can improve the clustering speed and accuracy of the clustering results when performing text clustering tasks.

[0054] Please see Figure 2 This is a flowchart illustrating another text clustering method provided in an embodiment of the present invention. The text clustering method includes the following steps:

[0055] S201: Obtain at least one text corpus.

[0056] S202: Vectorize each text corpus in at least one text corpus to obtain at least one text vector corresponding to at least one text corpus.

[0057] S203: Calculate the similarity of at least one text vector to obtain the text similarity matrix corresponding to the at least one text vector.

[0058] S204: Divide the text similarity matrix into K text similarity blocks, where the similarity of each text similarity block in the K text similarity blocks is greater than or equal to the similarity threshold, and the number of similarities is greater than or equal to the number of segments threshold, where K is an integer greater than or equal to 1.

[0059] The specific implementation methods of steps S201-S204 can be found in the specific implementation methods of steps S101-S104 in the foregoing embodiments, and will not be described in detail here.

[0060] S205: Calculate the overlap between any two text similarity blocks in K text similarity blocks.

[0061] It should be understood that the overlap is the degree of overlap between the text corpora contained in two text similarity blocks. For example, if the text corpora corresponding to text similarity block 1 include corpora 1, corpora 2, corpora 3, and corpora 5, and the text corpora corresponding to text similarity block 2 include corpora 1, corpora 2, corpora 4, corpora 5, and corpora 6, then the overlap is the ratio of the number of text corpora intersections (3) between text similarity block 1 and text similarity block 2 to the number of text corpora unions (6) between text similarity block 1 and text similarity block 2. The ratio is 0.5, that is, the overlap between text similarity block 1 and text similarity block 2 is 0.5.

[0062] S206: Determine N text similarity blocks from K text similarity blocks based on the overlap between any two text similarity blocks, wherein the overlap between each of the N text similarity blocks and any other text similarity block in the K text similarity blocks is less than or equal to a preset overlap threshold.

[0063] Where N is a positive integer less than or equal to K. The preset overlap threshold was calculated by the developers based on experimental data and can be adjusted accordingly based on specific application scenarios; no specific limit is set here.

[0064] The server selects N text similarity blocks from K text similarity blocks whose overlap with any other text similarity block is less than or equal to a preset overlap threshold. For example, if the preset overlap threshold is 0.3, and given text similarity blocks 1, 2, and 3, the calculated overlap between text similarity block 2 and text similarity block 1 is 0.2, the overlap between text similarity block 3 and text similarity block 1 is 0.1, and the overlap between text similarity block 3 and text similarity block 2 is 0.4, then text similarity block 3 is removed, and text similarity blocks 1 and 2 are retained. This method can improve the inter-class distance between different text clusters and improve the accuracy of clustering results.

[0065] S207: Determine the N text clusters corresponding to the N text similarity blocks.

[0066] The server determines the N text clusters corresponding to the N text similarity blocks based on the text corpus corresponding to the N text similarity blocks.

[0067] In one embodiment, the server obtains the target text vector corresponding to the similarity of each of the N text similarity blocks. Further, the server can use the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block to obtain N text clusters.

[0068] For example, the server segments the text similarity matrix row by row to obtain 6 text similarity blocks. Each text similarity block contains 6 similarities, each similarity corresponds to two target text vectors, and each target text vector corresponds to one text corpus. The server then obtains the target text vectors corresponding to each similarity in each text similarity block and uses the text corpus corresponding to each target text vector as the text corpus included in each text cluster, resulting in 6 text clusters as follows: Figure 3 As shown, module 31 contains the number of text corpora for each text cluster, and module 32 contains the text corpora included in each text cluster.

[0069] In one embodiment, the server can use the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block to obtain N text clusters. Specifically, the server uses the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block, and obtains the similarity between the text vectors corresponding to any two text corpora included in each text cluster. Further, the server can determine the target text corpus corresponding to each text cluster based on the similarity between the text vectors corresponding to the two text corpora, and then perform text corpus reduction on each text cluster based on the target text corpus to obtain N text clusters. This method ensures that any text corpus contained in each text cluster is similar to the other text corpora, increasing the intra-cluster similarity of each text cluster.

[0070] For example, after obtaining the target text vectors corresponding to the similarity of a certain text similarity block, the server obtains target text vectors a1, a2, a3, and a4. The text corpus 1, corpus 2, corpus 3, and corpus 4 corresponding to these target text vectors are taken as the text clusters corresponding to the text similarity block. The similarity a between target text vectors a1 and a2 is then obtained from the aforementioned text similarity matrix. 12 The similarity a between target text vector a1 and target text vector a3 13 The similarity a between target text vector a1 and target text vector a4 14 The similarity a between target text vector a2 and target text vector a3 23 The similarity a between target text vector a2 and target text vector a4 24 The similarity a between target text vector a3 and target text vector a4 34 The similarity matrix is ​​shown in the following formula:

[0071]

[0072] In this similarity matrix, the first row vector corresponds to the similarity between target text vector a1 and other target text vectors; the second row vector corresponds to the similarity between target text vector a2 and other target text vectors; the third row vector corresponds to the similarity between target text vector a3 and other target text vectors; and the fourth row vector corresponds to the similarity between target text vector a4 and other target text vectors. If the number of valid elements in the fourth row vector of the similarity matrix (i.e., the number of elements with similarity greater than or equal to the aforementioned similarity threshold) is less than the aforementioned segmentation threshold, then target text vector a4 is removed from the target text vectors a1, a2, a3, and a4 corresponding to the similarity in the text similarity block. That is, the text corpus 4 corresponding to target text vector a4 is removed from the text corpus corresponding to the text similarity block.

[0073] In one embodiment, the server can also obtain the degree of difference between the various text corpora included in any one of the N text clusters, and perform deduplication processing on the text corpora in any one text cluster based on the degree of difference between the various text corpora.

[0074] The difference between two text corpora can be measured by edit distance, which is used to measure the overlap between words in the two text corpora. Edit distance refers to the minimum number of single-character edit operations required to transform one word into another between two words. The value of edit distance ranges from [0,1]. An edit distance of 0 indicates a large difference between the two text corpora, while an edit distance of 1 indicates a small difference between the two text corpora.

[0075] The server can calculate the edit distance between the text corpora in each of the aforementioned N text clusters, that is, calculate the repetition between any two text corpora at the character (or word) level. If the edit distance between any two text corpora is greater than a preset edit distance threshold, it can be considered that one of the two text corpora is repeated by the other. The character lengths of the two text corpora are then calculated, and the text corpus with the larger character length is retained. The preset edit distance threshold was calculated by the developers based on experimental data and can be adjusted according to specific application scenarios; no specific limitation is made here. For example, text corpus 1 "Have you eaten?" and text corpus 2 "Have you eaten?" can be considered as text corpus 2 being repeated by text corpus 1. Since the character length of text corpus 1 is greater than the character length of text corpus 2, text corpus 1 is retained in this text cluster, and text corpus 2 is removed.

[0076] In an application scenario, such as Figure 4The illustration shows another text clustering method provided in this embodiment. In this method, the server acquires a large-scale short text (i.e., at least one text corpus mentioned above), performs word segmentation on each short text (i.e., each text corpus mentioned above), and performs part-of-speech tagging on each word. Then, the server can calculate the inverse document frequency (INF) value of each word in each short text, and characterize each short text (i.e., the sentence mentioned in the figure) based on the INF results and the INF values ​​of each word in each text corpus, obtaining at least one text vector. To improve server computation speed, the server employs a distributed computing method, such as... Figure 4 As shown, M machines (or application servers) are used to calculate the similarity between any two text vectors in at least one text vector, and the calculation results of each machine are summarized to construct a text similarity matrix corresponding to at least one text vector. Further, the server sorts the similarities in the text similarity matrix by row or column, and segments the text similarity matrix to obtain N similarity blocks (i.e., the N text similarity blocks mentioned in Example 2 above). Each of the N similarity blocks contains similarities greater than or equal to a similarity threshold, and the number of similarities is greater than or equal to a segmentation threshold. Therefore, the server can use P machines (or application servers) to determine the text cluster corresponding to each similarity block (i.e., ...) based on the N similarity blocks. Figure 4 The method involves clustering texts (as shown in the diagram) and calculating the difference between the text corpora within each text cluster corresponding to each similarity block (i.e., the edit distance mentioned earlier). Based on the difference between the text corpora, duplicate text corpora within any text cluster are removed. Thus, the server can aggregate the calculation results from P machines to obtain the aggregated result (i.e., the N text clusters mentioned in Example 2 above). This text clustering method improves the accuracy of the text clustering results and increases the speed of text clustering by utilizing distributed computing.

[0077] In this embodiment, the server acquires at least one text corpus and vectorizes each text corpus to obtain at least one text vector corresponding to the at least one text corpus. Further, the server can calculate the similarity of the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector, and segment the text similarity matrix to obtain K text similarity blocks. Each of the K text similarity blocks contains a similarity score greater than or equal to a similarity threshold, and the number of similar scores is greater than or equal to a segmentation number threshold. K is an integer greater than or equal to 1. Furthermore, the server calculates the overlap between any two text similarity blocks in the K text similarity blocks, and determines N text similarity blocks from the K text similarity blocks based on the overlap between any two text similarity blocks. Each of the N text similarity blocks has an overlap score less than or equal to a preset overlap threshold with any other text similarity block in the K text similarity blocks. Thus, the server determines N text clusters corresponding to the N text similarity blocks. This method filters the clustering results in text clustering tasks, increases the inter-cluster differences between text clusters, enhances intra-cluster similarity, and further improves the accuracy of text clustering results.

[0078] Based on the description of the above method embodiments, this invention also proposes a text clustering device. Please refer to... Figure 5 As shown, the text clustering device includes the following modules:

[0079] Acquisition module 50 is used to acquire at least one text corpus;

[0080] Processing module 51 is used to perform vectorization processing on each text corpus in the at least one text corpus to obtain at least one text vector corresponding to the at least one text corpus;

[0081] The processing module 51 is further configured to perform similarity calculation on the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector;

[0082] The processing module 51 is further configured to segment the text similarity matrix to obtain K text similarity blocks, and obtain N text clusters based on the K text similarity blocks. The similarity of each text similarity block in the K text similarity blocks is greater than or equal to a similarity threshold, and the number of similarities is greater than or equal to a segmentation number threshold. K is an integer greater than or equal to 1, and N is a positive integer less than or equal to K.

[0083] In one embodiment, the processing module 51 is specifically configured to perform word segmentation on each text corpus in the at least one text corpus to obtain each word included in each text corpus, and to perform part-of-speech tagging on each word; calculate the inverse document frequency value of each word in each text corpus; and perform vectorization processing on each text corpus based on the part-of-speech tagging results and the inverse document frequency value of each word in each text corpus to obtain at least one text vector corresponding to the at least one text corpus.

[0084] In one embodiment, the processing module 51 is specifically used to calculate the similarity between any two text vectors in the at least one text vector according to a distributed computing method; and to construct a text similarity matrix corresponding to the at least one text vector based on the similarity between the text vectors.

[0085] In one embodiment, the processing module 51 is specifically used to scan the text similarity matrix row by row to obtain at least one row vector included in the text similarity matrix; count the number of valid elements whose similarity is greater than or equal to a similarity threshold in each row vector; and use the row vectors whose number of valid elements is greater than or equal to a segmentation threshold as the text similarity matrix to obtain K text similarity blocks.

[0086] In one embodiment, the processing module 51 is specifically used to calculate the overlap between any two text similarity blocks in the K text similarity blocks; determine N text similarity blocks from the K text similarity blocks based on the overlap between any two text similarity blocks, wherein the overlap between each of the N text similarity blocks and any other text similarity block in the K text similarity blocks is less than or equal to a preset overlap threshold; and determine N text clusters corresponding to the N text similarity blocks.

[0087] In one embodiment, the processing module 51 is specifically used to obtain the target text vector corresponding to the similarity of each text similarity block in the N text similarity blocks; and to use the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block to obtain N text clusters.

[0088] In one embodiment, the processing module 51 is specifically configured to: use the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block; obtain the similarity between the text vectors corresponding to any two text corpora included in each text cluster; determine the target text corpus corresponding to each text cluster based on the similarity between the text vectors corresponding to any two text corpora; and reduce the text corpus of each text cluster based on the target text corpus to obtain N text clusters.

[0089] In one embodiment, the processing module 51 is further configured to obtain the degree of difference between the various text corpora included in any one of the N text clusters; and to perform deduplication processing on the text corpora in the arbitrary text cluster according to the degree of difference between the various text corpora.

[0090] It should be noted that the functions of each module of the text clustering device described in the embodiments of the present invention can be adjusted according to... Figure 1 or Figure 2 The specific implementation of the method in the embodiments described herein can be found in the following description. Figure 1 or Figure 2 The relevant descriptions of the method implementation examples will not be repeated here.

[0091] Based on the descriptions of the method and apparatus embodiments above, this invention also provides a server. Please refer to... Figure 6The server may include at least a processor 601, an input device 602, an output device 603, and a memory 604; wherein the processor 601, input device 602, output device 603, and memory 604 can be connected via a bus or other connection methods. The memory 604 is used to store computer programs, the computer programs including program instructions, and the processor 601 is used to execute the program instructions stored in the memory 604. The processor 601 (or CPU (Central Processing Unit)) is the computing and control core of the server, and it is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to realize the corresponding method flow or corresponding function in the above-described text clustering method embodiments. The processor 601 is configured to execute the following program instructions: acquire at least one text corpus; perform vectorization processing on each text corpus in the at least one text corpus to obtain at least one text vector corresponding to the at least one text corpus; calculate the similarity of the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector; segment the text similarity matrix to obtain K text similarity blocks, and obtain N text clusters based on the K text similarity blocks, wherein the similarity of each text similarity block in the K text similarity blocks is greater than or equal to a similarity threshold, and the number of similarities is greater than or equal to a segmentation number threshold, wherein K is an integer greater than or equal to 1, and N is a positive integer less than or equal to K.

[0092] In one embodiment, the processor 601 is specifically configured to perform word segmentation processing on each text corpus in the at least one text corpus to obtain each word included in each text corpus, and to perform part-of-speech tagging on each word; calculate the inverse document frequency value of each word in each text corpus; and perform vectorization processing on each text corpus based on the part-of-speech tagging results and the inverse document frequency value of each word in each text corpus to obtain at least one text vector corresponding to the at least one text corpus.

[0093] In one embodiment, the processor 601 is specifically configured to calculate the similarity between any two text vectors in the at least one text vector according to a distributed computing method; and to construct a text similarity matrix corresponding to the at least one text vector based on the similarity between the text vectors.

[0094] In one embodiment, the processor 601 is specifically configured to scan the text similarity matrix row by row to obtain at least one row vector included in the text similarity matrix; count the number of valid elements whose similarity is greater than or equal to a similarity threshold in each row vector; and use the row vectors whose number of valid elements is greater than or equal to a segmentation threshold as the text similarity matrix to obtain K text similarity blocks.

[0095] In one embodiment, the processor 601 is specifically configured to calculate the overlap between any two text similarity blocks in the K text similarity blocks; determine N text similarity blocks from the K text similarity blocks based on the overlap between any two text similarity blocks, wherein the overlap between each of the N text similarity blocks and any other text similarity block in the K text similarity blocks is less than or equal to a preset overlap threshold; and determine N text clusters corresponding to the N text similarity blocks.

[0096] In one embodiment, the processor 601 is specifically configured to obtain the target text vector corresponding to the similarity of each text similarity block in the N text similarity blocks; and to use the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block to obtain N text clusters.

[0097] In one embodiment, the processor 601 is specifically configured to: use the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block; obtain the similarity between the text vectors corresponding to any two text corpora included in each text cluster; determine the target text corpus corresponding to each text cluster based on the similarity between the text vectors corresponding to any two text corpora; and perform text corpus deletion on each text cluster based on the target text corpus to obtain N text clusters.

[0098] In one embodiment, the processor 601 is further configured to obtain the degree of difference between the various text corpora included in any one of the N text clusters; and to perform deduplication processing on the text corpora in the arbitrary text cluster according to the degree of difference between the various text corpora.

[0099] It should be understood that, in this embodiment of the invention, the processor 601 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0100] The memory 604 may include read-only memory and random access memory, and provides instructions and data to the processor 601. A portion of the memory 604 may also include non-volatile random access memory. For example, the memory 604 may also store device type information. The input device 602 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information), a microphone, a physical keyboard, etc., and the output device 603 may include a display (LCD, etc.), a speaker, etc.

[0101] In specific implementations, the processor 601, memory 604, input device 602, and output device 603 described in the embodiments of the present invention can execute the functions provided in the embodiments of the present invention. Figure 1 or Figure 2 The implementation described in the method embodiments can also be executed in accordance with the embodiments of the present invention. Figure 5 The implementation method of the described text clustering device will not be elaborated here.

[0102] In another embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, implement the embodiment of the present invention. Figure 1 or Figure 2The method described herein implements the aforementioned methods, wherein the computer-readable storage medium may be an internal storage unit of the server as described in any of the foregoing embodiments, such as a server hard drive or memory. The computer-readable storage medium may also be an external storage device of the server, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., provided on the server. Further, the computer-readable storage medium may include both internal storage units and external storage devices of the server. The computer-readable storage medium is used to store the computer program and other programs and data required by the server. The computer-readable storage medium may also be used to temporarily store data that has been output or will be output.

[0103] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.

[0104] The readable storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

Claims

1. A text clustering method, characterized in that, The method includes: Obtain at least one text corpus; Each text corpus in the at least one text corpus is vectorized to obtain at least one text vector corresponding to the at least one text corpus; The similarity of the at least one text vector is calculated to obtain the text similarity matrix corresponding to the at least one text vector; The text similarity matrix is ​​segmented to obtain K text similarity blocks, including: scanning the text similarity matrix row by row to obtain at least one row vector included in the text similarity matrix; counting the number of valid elements with a similarity greater than or equal to a similarity threshold in each row vector; and taking the row vectors with a number of valid elements greater than or equal to a segmentation threshold as the text similarity blocks, thus obtaining K text similarity blocks; or, scanning the text similarity matrix column by column to obtain at least one column vector included in the text similarity matrix; counting the number of valid elements with a similarity greater than or equal to a similarity threshold in each column vector; and taking the column vectors with a number of valid elements greater than or equal to a segmentation threshold as the text similarity blocks, thus obtaining K text similarity blocks, wherein the similarity of each of the K text similarity blocks is greater than or equal to the similarity threshold, and the number of similarities is greater than or equal to the segmentation threshold, and K is an integer greater than or equal to 1; Calculate the overlap between any two text similarity blocks in the K text similarity blocks; Based on the overlap between any two text similarity blocks, N text similarity blocks are determined from the K text similarity blocks. The overlap between each of the N text similarity blocks and any other text similarity block in the K text similarity blocks is less than or equal to a preset overlap threshold, where N is a positive integer less than or equal to K. Obtain the target text vector corresponding to the similarity of each of the N text similarity blocks, and use the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block; Obtain the similarity between the text vectors corresponding to any two text corpora included in each text cluster, and determine the target text corpus corresponding to each text cluster based on the similarity between the text vectors corresponding to any two text corpora. Based on the target text corpus, the text corpus of each text cluster is reduced to obtain N text clusters.

2. The method according to claim 1, characterized in that, The step of vectorizing each text corpus in the at least one text corpus to obtain at least one text vector corresponding to the at least one text corpus includes: Each text corpus in the at least one text corpus is segmented to obtain each segment of the text corpus, and each segment is tagged with part-of-speech tags. Calculate the inverse document frequency value of each word segment in each text corpus; Based on the part-of-speech tagging results and the inverse document frequency value of each word segment in each text corpus, each text corpus is vectorized to obtain at least one text vector corresponding to the at least one text corpus.

3. The method according to claim 1 or 2, characterized in that, The step of calculating the similarity of the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector includes: Calculate the similarity between any two text vectors in the at least one text vector using a distributed computing method; Based on the similarity between each text vector, construct a text similarity matrix corresponding to the at least one text vector.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the degree of difference between the various text corpora included in any one of the N text clusters; The text corpora in any one of the text clusters are deduplicated based on the degree of difference between the various text corpora.

5. A text clustering device, characterized in that, The device includes: The acquisition module is used to acquire at least one text corpus. The processing module is used to perform vectorization processing on each text corpus in the at least one text corpus to obtain at least one text vector corresponding to the at least one text corpus; The processing module is further configured to perform similarity calculation on the at least one text vector to obtain a text similarity matrix corresponding to the at least one text vector; The processing module is further configured to segment the text similarity matrix to obtain K text similarity blocks, where K is an integer greater than or equal to 1; In the process of segmenting the text similarity matrix to obtain K text similarity blocks, the processing module is specifically used to scan the text similarity matrix row by row to obtain at least one row vector included in the text similarity matrix, count the number of valid elements with a similarity greater than or equal to a similarity threshold in each row vector, and take the row vectors with a number of valid elements greater than or equal to a segmentation threshold as the text similarity blocks to obtain K text similarity blocks; or, to scan the text similarity matrix column by column to obtain at least one column vector included in the text similarity matrix, count the number of valid elements with a similarity greater than or equal to a similarity threshold in each column vector, and take the column vectors with a number of valid elements greater than or equal to a segmentation threshold as the text similarity blocks to obtain K text similarity blocks, wherein the similarity of each text similarity block in the K text similarity blocks is greater than or equal to the similarity threshold, and the number of similarities is greater than or equal to the segmentation threshold; The processing module is further configured to calculate the overlap between any two text similarity blocks in the K text similarity blocks, determine N text similarity blocks from the K text similarity blocks based on the overlap between any two text similarity blocks, wherein the overlap between each text similarity block in the N text similarity blocks and any other text similarity block in the K text similarity blocks is less than or equal to a preset overlap threshold, where N is a positive integer less than or equal to K, obtain the target text vector corresponding to the similarity included in each text similarity block in the N text similarity blocks, take the text corpus corresponding to the target text vector as the text cluster corresponding to each text similarity block, obtain the similarity between the text vectors corresponding to any two text corpora included in each text cluster, determine the target text corpus corresponding to each text cluster based on the similarity between the text vectors corresponding to any two text corpora, and perform text corpus deletion on each text cluster based on the target text corpus to obtain N text clusters.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of any one of claims 1-4.

Images