Text clustering method, device and equipment and storage medium thereof
By updating the dictionary, generating initial identifiers, and performing word segmentation in text clustering, and combining word frequency probability and weight for deduplication, the problems of slow computation speed and low accuracy in existing technologies are solved, and fast and accurate text clustering is achieved.
Patent Information
- Application Number
- CN202411072967.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-06
AI Technical Summary
Existing technologies are slow to compute and inaccurate in text clustering, especially the K-Shingle and K-means algorithms, which consume huge amounts of computational resources and produce inaccurate clustering results when processing large-scale text data.
By acquiring the text set to be clustered and caching it in the target database, updating the common word dictionary to record the latest word frequency probability and word weight, generating an initial clustering identifier for each text, performing word segmentation, and combining preliminary and secondary deduplication strategies to calculate the text entry repetition degree, the text clustering is finally achieved.
It achieves fast and accurate text clustering, reduces computational resource consumption, and improves clustering efficiency and accuracy.
Smart Images

Figure CN118939802A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and is applied to a scenario of clustering multi-source data texts, and in particular to a text clustering method, apparatus, device, and storage medium thereof. Background Art
[0002] There are many methods and steps for Internet text clustering technology. The core method is to use the text clustering method based on similarity. This method can be abstracted as the similarity matching problem between texts, which mainly solves the matching problem at the vocabulary level, or the similarity problem at the vocabulary level. For example, there are K-Shingle algorithm, conventional Jaccard similarity coefficient and K-means algorithm.
[0003] At present, the K-Shingle algorithm needs to generate a huge Shingle phrase library. When the number of texts and the length of texts are large, it takes a huge amount of time and space resources to calculate this phrase library, and the feature vector calculation of each document depends on this common phrase library. Therefore, it is difficult to fully parallelize the process of calculating the feature vector, resulting in a slow calculation speed. The K-means algorithm requires a predetermined number of clusters K value to be set, and the classification result is heavily dependent on the initialization of the cluster center. The result is not necessarily globally optimal, that is, the number of cluster categories needs to be set first, and the clustering result may be inaccurate. Therefore, the prior art still has the problem of slow calculation speed and inaccurate clustering in text clustering. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose a text clustering method, apparatus, device and storage medium thereof to solve the problems in the prior art of text clustering that the calculation speed is slow and the clustering is not accurate enough.
[0005] In order to solve the above technical problems, the present application embodiment provides a text clustering method, which adopts the following technical solution:
[0006] A text clustering method comprises the following steps:
[0007] Acquire the text set to be clustered and cache it in a target database, wherein a common word dictionary is preset in the target database, and the common word dictionary records the word frequency probability and word weight corresponding to all common words respectively;
[0008] Based on the changes in the text entries of the target database before and after caching, the latest word frequency probability and the latest word weight corresponding to all common words after the text set to be clustered is cached into the database are calculated;
[0009] Generating an initial clustering identifier for all the text items to be clustered in the text set to be clustered, wherein each text item to be clustered corresponds to a unique initial clustering identifier;
[0010] Perform word segmentation processing on all the text items to be clustered according to the common word dictionary to obtain word segmentation processing results;
[0011] Based on the word segmentation processing result, the initial clustering identifier and the preset preliminary duplicate detection strategy, the duplicate detection subset corresponding to each to-be-clustered text item is sequentially screened out from the to-be-clustered text set;
[0012] Calculate the repetition degree of each text item to be clustered and each text item in its corresponding duplicate detection subset by using the latest word frequency probability, the latest word weight and a preset secondary duplicate detection strategy;
[0013] Text clustering processing is performed on all the text items to be clustered according to the repetition degree to obtain a text clustering result.
[0014] In order to solve the above technical problems, the embodiment of the present application also provides a text clustering device, which adopts the following technical solution:
[0015] A text clustering device, comprising:
[0016] A text acquisition and caching module, used to acquire the text set to be clustered and cache it in a target database, wherein a common word dictionary is preset in the target database, and the common word dictionary records the word frequency probability and word weight corresponding to all common words respectively;
[0017] A dictionary updating module, for calculating the latest word frequency probability and the latest word weight corresponding to all common words after the text set to be clustered is cached into the database based on the text entry changes of the target database before and after caching;
[0018] An initial clustering identifier generating module, used for generating initial clustering identifiers for all the text items to be clustered in the text set to be clustered, wherein each text item to be clustered corresponds to a unique initial clustering identifier;
[0019] A word segmentation processing module, used to perform word segmentation processing on all the text items to be clustered according to the common word dictionary to obtain word segmentation processing results;
[0020] An initial duplicate detection module is used to sequentially filter out a duplicate detection subset corresponding to each to-be-clustered text entry from the to-be-clustered text set based on the word segmentation processing result, the initial clustering identifier and a preset preliminary duplicate detection strategy;
[0021] A secondary duplication detection module is used to calculate the duplication degree of each text item to be clustered and each text item in its corresponding duplication detection subset by using the latest word frequency probability, the latest word weight and a preset secondary duplication detection strategy;
[0022] The text clustering module is used to perform text clustering processing on all the text items to be clustered according to the repetition degree to obtain a text clustering result.
[0023] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0024] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned text clustering method when executing the computer-readable instructions.
[0025] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0026] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the text clustering method described above are implemented.
[0027] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0028] The text clustering method described in the embodiment of the present application is to obtain a text set to be clustered and cache it to a target database; update a common word dictionary to record the latest word frequency probability and the latest word weight corresponding to all common words; generate an initial clustering identifier for all text items to be clustered in the text set to be clustered; perform word segmentation processing on all text items to be clustered according to the common word dictionary to obtain a word segmentation processing result; based on the word segmentation processing result, the initial clustering identifier and the preset preliminary duplicate judgment strategy, select the duplicate judgment subset corresponding to each text item to be clustered from the text set to be clustered in turn; calculate the repetition of each text item to be clustered and each text item in its corresponding duplicate judgment subset through the latest word frequency probability, the latest word weight and the preset secondary duplicate judgment strategy; perform text clustering processing on all text items to be clustered according to the repetition to obtain a text clustering result. The method first performs preliminary duplicate judgment through the word segmentation processing result, and then performs secondary duplicate judgment in combination with the latest word frequency probability and the latest word weight, thereby realizing fast and accurate clustering of the text to be clustered. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0030] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0031] Figure 2 is a flowchart of an embodiment of a text clustering method according to the present application;
[0032] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;
[0033] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;
[0034] Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 402 is shown;
[0035] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 206 is shown;
[0036] Figure 7 It is a graphical schematic diagram of the path planning algorithm used in this embodiment to dynamically plan the path required from the upper left corner to the lower right corner of the word segmentation mapping matrix;
[0037] Figure 8 is a structural schematic diagram of an embodiment of a text clustering device according to the present application;
[0038] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0040] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0042] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0043] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0044] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0045] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0046] It should be noted that the text clustering method provided in the embodiment of the present application is generally executed by a server, and accordingly, the text clustering device is generally arranged in the server.
[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0048] Continue to refer Figure 2 , shows a flow chart of an embodiment of a text clustering method according to the present application. The text clustering method comprises the following steps:
[0049] Step 201 , obtaining a text set to be clustered, and caching it in a target database, wherein a common word dictionary is preset in the target database, and the common word dictionary records the word frequency probabilities and word weights corresponding to all common words.
[0050] In this embodiment, before executing the step of obtaining the text set to be clustered and caching it in the target database, the method also includes: using a preset preprocessing method to preprocess all the text items to be clustered in the text set to be clustered, wherein the preprocessing method includes noise data cleaning, unification of Chinese and English text languages, removal of punctuation, deletion of unnecessary spaces, and unification of text font formats; counting the number of text items contained in the target database, and counting the number of text items corresponding to all common words in the target database, to obtain a first statistical result.
[0051] In this embodiment, the step of obtaining the set of texts to be clustered and caching them in the target database specifically includes: sending a request for obtaining the texts to be clustered to a multi-source data terminal, the multi-source data terminal responds to the request for obtaining the texts to be clustered, and sends the texts to be clustered to a unique receiving terminal, at which time the multi-source data terminal can be multiple databases, data warehouses, or cloud databases, etc., or the texts to be clustered are captured from the multi-source data terminal through a preset page crawler component, at which time the multi-source data terminal can be a web browsing terminal with multiple different IP addresses, and then the texts to be clustered obtained from the multi-source data terminals are stored in the same set to generate a set of texts to be clustered, and the set of texts to be clustered is cached in the target database to facilitate subsequent calculations and clustering processing.
[0052] In this embodiment, after executing the step of obtaining the text set to be clustered and caching it in the target database, the method also includes: counting the number of text entries to be clustered in the text set to be clustered, and counting the number of text entries to be clustered corresponding to all common words in the target database, to obtain a second statistical result.
[0053] By obtaining the first statistical result and the second statistical result, it is convenient to subsequently update the common word dictionary, and the latest word frequency probability and the latest word weight corresponding to all common words are recorded in combination with the first statistical result and the second statistical result, so that as the new text set to be clustered is cached into the database, relevant updates are carried out, thereby ensuring the accuracy of subsequent text clustering.
[0054] Step 202, based on the text entry changes of the target database before and after caching, calculate the latest word frequency probability and the latest word weight corresponding to all common words after the text set to be clustered is cached into the database.
[0055] In this embodiment, the step of calculating the latest word frequency probabilities and latest word weights corresponding to all common words after the text set to be clustered is cached into the database based on the changes in text entries in the target database before and after caching, specifically includes: based on the statistical results of the number of text entries contained in the target database before and after caching, updating the common word dictionary, and calculating the latest word frequency probabilities and latest word weights corresponding to all common words after the text set to be clustered is cached into the database.
[0056] In this embodiment, the step of calculating the latest word frequency probability and the latest word weight corresponding to all common words after the text set to be clustered is cached and stored in the library specifically includes:
[0057] Based on the first statistical result, the second statistical result and a preset word frequency probability formula, the latest word frequency probabilities corresponding to all common words in the common word dictionary are calculated, wherein the preset word frequency probability formula is:
[0058]
[0059] a represents the identification information of the current common word, P a represents the latest word frequency probability of the current common words, M1 represents the number of text items contained in the target database before the text set to be clustered is cached in the target database, M2 represents the number of text items to be clustered in the text set to be clustered, N1 represents the number of text items in the target database containing the current common words before the text set to be clustered is cached in the target database, and N2 represents the number of text items to be clustered containing the current common words in the text set to be clustered;
[0060] According to the preset formula for the relationship between word frequency probability and word weight, the latest word weights corresponding to all common words in the common word dictionary are calculated, wherein the preset formula for the relationship between word frequency probability and word weight is:
[0061]
[0062] a represents the identification information of the current common word, Pa Indicates the latest frequency probability of the current common words, and 0 <P a <1, b means that the numerical parameter can be set, and 0 <b<1,ω a represents the word weight of the current common word, and ω a >0.
[0063] In this embodiment, the step of updating the common word dictionary also includes: counting the total number of times that specified words outside the common word dictionary appear in the target database according to a preset time interval or the cumulative number of times they are entered into the database, wherein the specified words include the latest words added by the operator through human-computer interaction, such as the latest network words, new buzzwords, etc.; judging whether the total number of times meets a preset number threshold; if the total number of times meets the preset number threshold, entering the specified word as a new common word into the common word dictionary; if the total number of times does not meet the preset number threshold, there is no need to enter the specified word as a new common word into the common word dictionary.
[0064] This application facilitates continuous enrichment of the common word dictionary by counting the total number of times the specified words outside the common word dictionary appear in the target database, and achieves continuous updating of the common word dictionary, further ensuring the accuracy of subsequent text clustering.
[0065] Step 203: generating initial clustering identifiers for all the text items to be clustered in the text set to be clustered, wherein each text item to be clustered corresponds to a unique initial clustering identifier.
[0066] In this embodiment, the initial clustering identifiers are generated for all the text items to be clustered in the text set to be clustered. Specifically, a preset identifier generation component is used to generate initial clustering identifiers with a consistent format for all the text items to be clustered in the text set to be clustered.
[0067] By generating initial clustering identifiers for all the text items to be clustered in the text set to be clustered, and each text item to be clustered corresponds to a unique initial clustering identifier, it is convenient to accurately identify and retrieve the corresponding text item to be clustered according to the initial clustering identifier during subsequent clustering operations.
[0068] Step 204: perform word segmentation processing on all the text items to be clustered according to the common word dictionary to obtain word segmentation processing results.
[0069] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes:
[0070] Step 301, importing the common word dictionary as a word segmentation basis into a preset word segmentation component, wherein the preset word segmentation component includes a subword-based word segmentation processing component;
[0071] In the embodiment of the present application, the preset word segmentation component can be divided into a word segmentation component based on semantic features, a word segmentation component based on fixed character length, and a word segmentation processing component based on subwords according to different word segmentation bases. The word segmentation component based on semantic features needs to pre-build a word semantic feature analysis library, and then segment the words by identifying the semantic features of each word in the text entry to be clustered when used. The word segmentation is relatively accurate, but it seriously affects the processing efficiency; the word segmentation component based on fixed character length generally sets a fixed word segmentation length to perform word segmentation on the text entry to be clustered. Although this method is relatively fast in word segmentation processing, it is not very suitable when encountering long and short words. For example: the fixed word segmentation length is 2 characters. When the current text entry to be clustered includes proverbs or multi-word idioms, it is obvious that this word segmentation method is not suitable enough; the word segmentation processing component based on subwords generally uses the common words in the common word dictionary as the word segmentation basis, which satisfies the requirements of fast word segmentation and is no longer limited to a fixed length, and is more applicable.
[0072] Step 302: input all the text items to be clustered as texts to be segmented into the segmentation component in sequence, and obtain the segmentation result corresponding to each text item to be clustered output by the segmentation component;
[0073] Step 303 , sorting the word segmentation results corresponding to each text item to be clustered according to the order in which each word segmentation appears in the corresponding text item to be clustered, and obtaining an ordered word segmentation array corresponding to each text item to be clustered.
[0074] By sorting the word segmentation results corresponding to each text entry to be clustered in the order in which each word segment appears in the corresponding text entry to be clustered, an ordered word segmentation array corresponding to each text entry to be clustered is obtained, so that a word segmentation mapping matrix can be constructed according to the ordered word segmentation array, so as to facilitate the subsequent dynamic path planning from the upper left corner to the lower right corner of the word segmentation mapping matrix in combination with the word segmentation mapping matrix and the path planning algorithm, and compare the repetition of two text entries to be clustered.
[0075] Step 205 , based on the word segmentation processing result, the initial clustering identifier and the preset preliminary duplicate detection strategy, filter out the duplicate detection subset corresponding to each to-be-clustered text item from the to-be-clustered text set in turn.
[0076] Continue to refer Figure 4 , Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes:
[0077] Step 401, based on the ordered segmentation array corresponding to each text to be clustered, counting the number of segmentations corresponding to each text item to be clustered;
[0078] Step 402 , according to the number of segmented words corresponding to each text item to be clustered, the initial clustering identifier and the preliminary duplicate detection strategy, filter out the duplicate detection subset corresponding to each text item to be clustered.
[0079] Continue to refer Figure 5 , Figure 5 yes Figure 4 The flowchart of a specific embodiment of step 402 shown includes:
[0080] Step 501, selecting different text items to be clustered from the text set to be clustered as reference text items in sequence according to the initial clustering identifier;
[0081] Step 502: selecting other text items to be clustered from the remaining text items to be clustered in turn as comparison text items according to the initial clustering identifier;
[0082] Step 503, calculating the difference between the number of segmented words in the reference text entry and the number of segmented words in the comparison text entry;
[0083] Step 504, determining whether the difference is less than a preset positive integer value;
[0084] Step 505: if the difference is greater than or equal to a preset positive integer value, re-screen the comparison text entries and execute steps 503 to 504 again;
[0085] In this embodiment, the positive integer value may be a fixed value or a variable value. When it is a variable value, the positive integer value increases in a certain positive proportion with the increase in the number of segmentations of the reference text entry. For example, when the number of segmentations is 10, the positive integer value is 2, when the number of segmentations is 15, the positive integer value is 4, and when the number of segmentations is 30, the positive integer value is 6.
[0086] Step 506, if the difference is less than a preset positive integer value, adding the comparison text entry to the duplicate determination subset corresponding to the reference text entry;
[0087] Step 507, repeating steps 501 to 506 until all the text items to be clustered in the text set to be clustered have become the benchmark text items, and all the benchmark text items have screened out the corresponding duplicate judgment subsets, then stopping the execution of steps 501 to 506.
[0088] In this embodiment, the preliminary duplicate detection strategy is to calculate the difference between the number of word segments of the benchmark text entry and the number of word segments of the comparison text entry; determine whether the difference is less than a preset positive integer value; if the difference is not less than the preset positive integer value, re-screen the comparison text entries and execute steps 503 to 504 again; if the difference is less than the preset positive integer value, add the comparison text entry to the duplicate detection subset corresponding to the benchmark text entry.
[0089] First, by comparing the number of segmented words of each text item to be clustered, the text set to be clustered is sorted, and the duplicate detection subsets corresponding to each text item to be clustered are screened out. That is, the text set to be clustered is preliminarily clustered based on the difference in the number of segmented words, and the duplicate detection subsets corresponding to each text item to be clustered after the preliminary clustering are obtained, which avoids the subsequent segmentation mapping matrix from being too thin and long, and ensures that the subsequent constructed segmentation mapping matrix is closer to a square, so as to quickly and accurately cluster the text set to be clustered, and to a certain extent reduces the consumption of the subsequent segmentation mapping matrix construction.
[0090] Step 206 , calculating the repetition degree between each text item to be clustered and each text item in its corresponding duplicate detection subset by using the latest word frequency probability, the latest word weight and a preset secondary duplicate detection strategy.
[0091] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 206 shown includes:
[0092] Step 601, selecting any text item to be clustered from the text set to be clustered as a first text item according to the initial clustering identifier, and
[0093] Step 602, selecting any text item from the duplicate determination subset corresponding to the first text item as the second text item;
[0094] Step 603, obtaining the ordered word segmentation array corresponding to the first text item as a first ordered word segmentation array, and obtaining the ordered word segmentation array corresponding to the second text item as a second ordered word segmentation array;
[0095] Step 604: construct a word segmentation mapping matrix according to the first ordered word segmentation array and the second ordered word segmentation array;
[0096] Step 605, respectively calculating the path calculation weight corresponding to each segmentation in the first ordered segmentation array and the second ordered segmentation array according to the latest word frequency probability corresponding to each segmentation, the latest word weight corresponding to each segmentation, and a preset path calculation weight algorithm;
[0097] In this embodiment, the step of respectively calculating the path calculation weights corresponding to each segmentation in the first ordered segmentation array and the second ordered segmentation array according to the latest word frequency probability corresponding to each segmentation, the latest word weight corresponding to each segmentation, and a preset path calculation weight algorithm specifically includes:
[0098] Use the preset path calculation weight algorithm:
[0099]
[0100] Calculate the path calculation weight algorithm corresponding to each word in the first ordered word segmentation array and the second ordered word segmentation array, where a represents the identification information of the current word segmentation, K a Indicates the word calculation weight of the current segmentation, P a Indicates the latest word frequency probability of the current word segmentation, and 0 <P a <1,ω a represents the word weight of the current word segmentation, and ω a >0, c represents the correction coefficient.
[0101] Step 606, dynamically planning the path required from the upper left corner to the lower right corner of the word segmentation mapping matrix using a path planning algorithm, and calculating the path weights corresponding to all paths based on the path calculation weights corresponding to each word in the first ordered word segmentation array and the second ordered word segmentation array;
[0102] Continue to refer Figure 7 , Figure 7A graphical schematic diagram of the path planning algorithm used in this embodiment to dynamically plan the required path from the upper left corner to the lower right corner of the word segmentation mapping matrix, wherein A, B, C, D, and E in the horizontal columns of the word segmentation mapping matrix represent each word in the first ordered word segmentation array, and the word segmentation sequence contained in the first ordered word segmentation array is [A, B, C, C, B, D, B, D, E]; A, B, C, D, E, and F in the vertical columns of the word segmentation mapping matrix represent each word in the second ordered word segmentation array, and the word segmentation sequence contained in the second ordered word segmentation array is [A, B, C, D, B, D, F, D, E, E]; the word segmentation mapping The shaded part in the matrix indicates that the word segmentation corresponding to the horizontal column and the vertical column is the same word segmentation, and the shaded part corresponds to the oblique line from the upper left corner to the lower right corner of the cell; it is stipulated that the dynamic path can select the vertical edge, oblique line and horizontal edge of the cell in the word segmentation mapping matrix from top to bottom and from left to right, so as to plan the path required from the upper left corner to the lower right corner of the word segmentation mapping matrix; it is also stipulated in advance that when the path segment is a vertical part, the length of the vertical path segment is the path calculation weight of the corresponding word segmentation in the vertical column; when the path segment is a horizontal part, the length of the vertical path segment is the path calculation weight of the corresponding word segmentation in the horizontal column; when the path segment is a diagonal line in the shaded part, the length of the diagonal line path segment is set to 0, Figure 7 Three dynamic planning paths are shown in FIG. 7a , 7b and 7c .
[0103] In this embodiment, the step of using a path planning algorithm to dynamically plan the path required from the upper left corner to the lower right corner of the word segmentation mapping matrix, and calculating the path weights corresponding to all paths based on the path calculation weights corresponding to each word in the first ordered word segmentation array and the second ordered word segmentation array, specifically includes: arbitrarily selecting a path as the current path; identifying all the word segmentations corresponding to the vertical part path and the horizontal part path in the current path; accumulating the path calculation weights of all the word segmentations to obtain an accumulated value; and using the accumulated value as the path weight corresponding to the current path.
[0104] Specifically, for example, in the path diagram shown in 7a, the segmentation words corresponding to the vertical part of the path include D, F and E, and the segmentation words corresponding to the horizontal part of the path include C and B. The path calculation weights of B, C, D, E and F are accumulated to obtain the path weight corresponding to the current path; for another example, in the path diagram shown in 7b, the segmentation words corresponding to the vertical part of the path include F, D and E, and the segmentation words corresponding to the horizontal part of the path include C and B. The path calculation weights of B, C, D, E and F are accumulated to obtain the path weight corresponding to the current path; in fact, the path weights calculated in 7a and 7b are the same. Continuing to refer to 7c, the path diagram shown in 7c, the segmentation words corresponding to the vertical part of the path include B, C, D, D, F and E, and the segmentation words corresponding to the horizontal part of the path include C, C, B, B and D. The path calculation weights of B, C, D, D, F, E, C, C, B, B and D are accumulated to obtain the path weight corresponding to the current path. Obviously, Figure 7 The path weight of the path shown in 7c is greater than the path weights of the paths shown in 7a and 7b.
[0105] Step 607: Filter out the minimum path weight from the path weights corresponding to all the paths, and calculate the repetition of the first text entry and the second text entry in combination with a preset repetition calculation formula.
[0106] In this embodiment, the step of selecting the minimum path weight from the path weights corresponding to all paths, and calculating the repetition of the first text item and the second text item in combination with a preset repetition calculation formula specifically includes:
[0107] The preset repeatability calculation formula is:
[0108]
[0109] Calculate the repetition degree of the first text entry and the second text entry, where m represents the total number of words in the ordered word segmentation array corresponding to the first text entry, n represents the total number of words in the ordered word segmentation array corresponding to the second text entry, l represents the minimum path weight, and min(m,n) represents selecting a relatively small value from m and n.
[0110] Continuing with the above embodiment, since the word sequence contained in the first ordered word segmentation array is [A, B, C, C, B, D, B, D, E], the total number of words in the first ordered word segmentation array is 9. Similarly, the word sequence contained in the second ordered word segmentation array is [A, B, C, D, B, D, F, D, E, E], the total number of words in the second ordered word segmentation array is 10, that is, the values of m and n are 9 and 10 respectively, and the value of min(m, n) is 9. Correspondingly, the smaller the value of l, the larger the value of the repetition r.
[0111] Step 207: Perform text clustering processing on all the text items to be clustered according to the repetition degree to obtain a text clustering result.
[0112] In this embodiment, the step of performing text clustering processing on all the text entries to be clustered according to the repetition degree to obtain the text clustering result specifically includes: identifying whether the repetition degree exceeds a preset repetition degree threshold; if so, classifying the second text entry into the text clustering set where the first text entry is located; if not, continuing to select other text entries from the duplicate judgment subset corresponding to the first text entry as the second text entry, and continuing to execute steps 603 to 607.
[0113] By combining the path planning algorithm, the word segmentation mapping matrix and the repetition calculation formula, the repetition between each text entry to be clustered and each text entry in its corresponding duplicate judgment subset is calculated, and text clustering processing is performed on all the text entries to be clustered based on the repetition to obtain the text clustering result, thereby ensuring the accuracy of text clustering.
[0114] The present application obtains a set of texts to be clustered and caches it in a target database; updates a dictionary of common words, and records the latest word frequency probabilities and latest word weights corresponding to all common words; generates an initial clustering identifier for all text items to be clustered in the set of texts to be clustered; performs word segmentation processing on all text items to be clustered according to the dictionary of common words to obtain word segmentation processing results; based on the word segmentation processing results, the initial clustering identifier and the preset preliminary duplicate judgment strategy, selects the duplicate judgment subset corresponding to each text item to be clustered from the set of texts to be clustered in turn; calculates the repetition of each text item to be clustered and each text item in its corresponding duplicate judgment subset through the latest word frequency probability, the latest word weight and the preset secondary duplicate judgment strategy; performs text clustering processing on all text items to be clustered according to the repetition to obtain text clustering results. The method first performs preliminary duplicate judgment through the word segmentation processing results, and then performs secondary duplicate judgment in combination with the latest word frequency probability and the latest word weight, thereby realizing fast and accurate clustering of the texts to be clustered.
[0115] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0116] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, large text clustering technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0117] In the embodiment of the present application, a text set to be clustered is obtained and cached to a target database; a common word dictionary is updated to record the latest word frequency probability and the latest word weight corresponding to all common words; an initial clustering identifier is generated for all text items to be clustered in the text set to be clustered; all text items to be clustered are segmented according to the common word dictionary to obtain a word segmentation result; based on the word segmentation result, the initial clustering identifier and the preset preliminary duplicate judgment strategy, the duplicate judgment subset corresponding to each text item to be clustered is screened out from the text set to be clustered in turn; the repetition of each text item to be clustered and each text item in its corresponding duplicate judgment subset is calculated through the latest word frequency probability, the latest word weight and the preset secondary duplicate judgment strategy; all text items to be clustered are clustered according to the repetition to obtain a text clustering result. The method first performs a preliminary duplicate judgment through the word segmentation result, and then performs a secondary duplicate judgment in combination with the latest word frequency probability and the latest word weight, thereby realizing fast and accurate clustering of the text to be clustered.
[0118] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of a text clustering device, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0119] like Figure 8 As shown, the text clustering device 800 described in this embodiment includes: a text acquisition and cache module 801, a dictionary update module 802, an initial clustering identifier generation module 803, a word segmentation processing module 804, a primary duplicate detection module 805, a secondary duplicate detection module 806 and a text clustering module 807. Among them:
[0120] The text acquisition and caching module 801 is used to acquire the text set to be clustered and cache it in a target database, wherein a common word dictionary is preset in the target database, and the common word dictionary records the word frequency probability and word weight corresponding to all common words respectively;
[0121] A dictionary updating module 802 is used to calculate the latest word frequency probability and the latest word weight corresponding to all common words after the text set to be clustered is cached into the database based on the changes of text entries in the target database before and after caching;
[0122] An initial clustering identifier generating module 803 is used to generate initial clustering identifiers for all the text items to be clustered in the text set to be clustered, wherein each text item to be clustered corresponds to a unique initial clustering identifier;
[0123] A word segmentation processing module 804 is used to perform word segmentation processing on all the text items to be clustered according to the common word dictionary to obtain a word segmentation processing result;
[0124] The initial duplicate detection module 805 is used to sequentially select the duplicate detection subset corresponding to each to-be-clustered text item from the to-be-clustered text set based on the word segmentation processing result, the initial clustering identifier and the preset preliminary duplicate detection strategy;
[0125] The secondary duplication detection module 806 is used to calculate the duplication degree between each text item to be clustered and each text item in its corresponding duplication detection subset by using the latest word frequency probability, the latest word weight and a preset secondary duplication detection strategy;
[0126] The text clustering module 807 is used to perform text clustering processing on all the text items to be clustered according to the repetition degree to obtain a text clustering result.
[0127] The present application obtains a set of texts to be clustered and caches it in a target database; updates a dictionary of common words, and records the latest word frequency probabilities and latest word weights corresponding to all common words; generates an initial clustering identifier for all text items to be clustered in the set of texts to be clustered; performs word segmentation processing on all text items to be clustered according to the dictionary of common words to obtain word segmentation processing results; based on the word segmentation processing results, the initial clustering identifier and the preset preliminary duplicate judgment strategy, selects the duplicate judgment subset corresponding to each text item to be clustered from the set of texts to be clustered in turn; calculates the repetition of each text item to be clustered and each text item in its corresponding duplicate judgment subset through the latest word frequency probability, the latest word weight and the preset secondary duplicate judgment strategy; performs text clustering processing on all text items to be clustered according to the repetition to obtain text clustering results. The method first performs preliminary duplicate judgment through the word segmentation processing results, and then performs secondary duplicate judgment in combination with the latest word frequency probability and the latest word weight, thereby realizing fast and accurate clustering of the texts to be clustered.
[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0129] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0130] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.
[0131] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 9 having components 9a-9c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.
[0132] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0133] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 9a can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as a computer-readable instruction of a text clustering method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.
[0134] The processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other text clustering chips in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run the computer-readable instructions or process data stored in the memory 9a, such as computer-readable instructions for running the text clustering method.
[0135] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.
[0136] The computer device proposed in this embodiment belongs to the field of data processing technology and is applied to the scene of clustering multi-source data text. The present application obtains a text set to be clustered and caches it to a target database; updates a common word dictionary and records the latest word frequency probability and the latest word weight corresponding to all common words; generates an initial clustering identifier for all text items to be clustered in the text set to be clustered; performs word segmentation processing on all text items to be clustered according to the common word dictionary to obtain a word segmentation processing result; based on the word segmentation processing result, the initial clustering identifier and the preset preliminary weight judgment strategy, the weight judgment subset corresponding to each text item to be clustered is screened out from the text set to be clustered in turn; the repetition of each text item to be clustered and each text item in its corresponding weight judgment subset is calculated through the latest word frequency probability, the latest word weight and the preset secondary weight judgment strategy; all text items to be clustered are subjected to text clustering processing according to the repetition to obtain a text clustering result. The method first performs preliminary weight judgment through the word segmentation processing result, and then performs secondary weight judgment in combination with the latest word frequency probability and the latest word weight, thereby realizing fast and accurate clustering of the text to be clustered.
[0137] The present application also provides another implementation, that is, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the text clustering method as described above.
[0138] The computer-readable storage medium proposed in this embodiment belongs to the field of data processing technology and is applied to the scenario of clustering multi-source data text. This application obtains a set of texts to be clustered and caches it to a target database; updates a dictionary of common words and records the latest word frequency probabilities and latest word weights corresponding to all common words; generates an initial clustering identifier for all text items to be clustered in the set of texts to be clustered; performs word segmentation processing on all text items to be clustered according to the dictionary of common words to obtain word segmentation processing results; based on the word segmentation processing results, the initial clustering identifier and the preset preliminary duplicate judgment strategy, selects the duplicate judgment subset corresponding to each text item to be clustered from the set of texts to be clustered in turn; calculates the repetition of each text item to be clustered and each text item in its corresponding duplicate judgment subset through the latest word frequency probability, the latest word weight and the preset secondary duplicate judgment strategy; performs text clustering processing on all text items to be clustered according to the repetition to obtain text clustering results. The method first uses the word segmentation processing results to make a preliminary duplicate judgment, and then combines the latest word frequency probability and the latest word weight to make a secondary duplicate judgment, thereby achieving fast and accurate clustering of clustered texts.
[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0140] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A text clustering method, It is characterized in that The steps include: Acquire the text set to be clustered and cache it in a target database, wherein a common word dictionary is preset in the target database, and the common word dictionary records the word frequency probability and word weight corresponding to all common words respectively; Based on the changes in the text entries of the target database before and after caching, the latest word frequency probability and the latest word weight corresponding to all common words after the text set to be clustered is cached into the database are calculated; Generating an initial clustering identifier for all the text items to be clustered in the text set to be clustered, wherein each text item to be clustered corresponds to a unique initial clustering identifier; Perform word segmentation processing on all the text items to be clustered according to the common word dictionary to obtain word segmentation processing results; Based on the word segmentation processing result, the initial clustering identifier and the preset preliminary duplicate detection strategy, the duplicate detection subset corresponding to each to-be-clustered text item is sequentially screened out from the to-be-clustered text set; Calculate the repetition degree of each text item to be clustered and each text item in its corresponding duplicate detection subset by using the latest word frequency probability, the latest word weight and a preset secondary duplicate detection strategy; Text clustering processing is performed on all the text items to be clustered according to the repetition degree to obtain a text clustering result.
2. The text clustering method according to claim 1, It is characterized in that Before executing the step of obtaining the text set to be clustered and caching it in the target database, the method further includes: Preprocessing all the text items to be clustered in the text set to be clustered by using a preset preprocessing method, wherein the preprocessing method includes cleaning noise data, unifying the Chinese and English text languages, removing punctuation marks, deleting unnecessary spaces, and unifying the text font format; Counting the number of text entries contained in the target database, and counting the number of text entries corresponding to all common words in the target database, to obtain a first statistical result; After executing the step of obtaining the text set to be clustered and caching it in the target database, the method further includes: The number of text items to be clustered in the text set to be clustered is counted, and the number of text items to be clustered corresponding to all common words in the target database is counted to obtain a second statistical result.
3. The text clustering method according to claim 2, It is characterized in that The step of calculating the latest word frequency probability and the latest word weight corresponding to all common words after the text set to be clustered is cached and stored in the database specifically includes: Based on the first statistical result, the second statistical result and a preset word frequency probability formula, the latest word frequency probabilities corresponding to all common words in the common word dictionary are calculated, wherein the preset word frequency probability formula is: a represents the identification information of the current common word, P a Represents the latest frequency probability of the current commonly used words, M 1 represents the number of text entries contained in the target database before the text set to be clustered is cached in the target database, M 2 N represents the number of text items to be clustered in the text set to be clustered. 1 N represents the number of text entries containing the current common words in the target database before the text set to be clustered is cached in the target database. 2 Indicates the number of text items to be clustered that contain the current common words in the text set to be clustered; According to the preset formula for the relationship between word frequency probability and word weight, the latest word weights corresponding to all common words in the common word dictionary are calculated, wherein the preset formula for the relationship between word frequency probability and word weight is: a represents the identification information of the current common word, P a Indicates the latest frequency probability of the current common words, and 0 <P a <1, b means that the numerical parameter can be set, and 0 <b<1,ω a represents the word weight of the current common word, and ω a >0.
4. The text clustering method according to claim 1, It is characterized in that The step of performing word segmentation processing on all the text items to be clustered according to the common word dictionary to obtain the word segmentation processing result specifically includes: Importing the common word dictionary as a word segmentation basis into a preset word segmentation component; Inputting all the text items to be clustered as texts to be segmented into the word segmentation component in sequence, and obtaining the segmentation results corresponding to each text item to be clustered output by the word segmentation component; According to the order in which each word segmentation appears in the corresponding text item to be clustered, the word segmentation results corresponding to each text item to be clustered are sorted in an ordered manner to obtain an ordered word segmentation array corresponding to each text item to be clustered; The step of sequentially selecting a duplicate detection subset corresponding to each to-be-clustered text item from the to-be-clustered text set based on the word segmentation processing result, the initial clustering identifier and the preset preliminary duplicate detection strategy specifically includes: Based on the ordered word segmentation array corresponding to each text to be clustered, count the number of word segments corresponding to each text item to be clustered; According to the number of segmented words corresponding to each text item to be clustered, the initial clustering identifier and the preliminary duplicate determination strategy, the duplicate determination subset corresponding to each text item to be clustered is screened out.
5. The text clustering method according to claim 4, It is characterized in that The step of screening out the duplicate detection subset corresponding to each text item to be clustered according to the number of segmented words corresponding to each text item to be clustered, the initial clustering identifier and the preliminary duplicate detection strategy specifically includes: Step 501, selecting different text items to be clustered from the text set to be clustered as reference text items in sequence according to the initial clustering identifier; Step 502: selecting other text items to be clustered from the remaining text items to be clustered in turn as comparison text items according to the initial clustering identifier; Step 503, calculating the difference between the number of segmented words in the reference text entry and the number of segmented words in the comparison text entry; Step 504, determining whether the difference is less than a preset positive integer value; Step 505: if the difference is greater than or equal to a preset positive integer value, re-screen the comparison text entries and execute steps 503 to 504 again; Step 506, if the difference is less than a preset positive integer value, adding the comparison text entry to the duplicate determination subset corresponding to the reference text entry; Step 507, repeating steps 501 to 506 until all the text items to be clustered in the text set to be clustered have become the reference text items, and all the reference text items have filtered out the corresponding duplicate determination subsets.
6. The text clustering method according to claim 1, It is characterized in that The step of calculating the repetition degree between each text item to be clustered and each text item in its corresponding duplicate detection subset by using the latest word frequency probability, the latest word weight and a preset secondary duplicate detection strategy specifically includes: Step 601, selecting any text item to be clustered from the text set to be clustered as a first text item according to the initial clustering identifier; Step 602, selecting any text item from the duplicate determination subset corresponding to the first text item as the second text item; Step 603, obtaining the ordered word segmentation array corresponding to the first text item as a first ordered word segmentation array, and obtaining the ordered word segmentation array corresponding to the second text item as a second ordered word segmentation array; Step 604: construct a word segmentation mapping matrix according to the first ordered word segmentation array and the second ordered word segmentation array; Step 605, respectively calculating the path calculation weight corresponding to each segmentation in the first ordered segmentation array and the second ordered segmentation array according to the latest word frequency probability corresponding to each segmentation, the latest word weight corresponding to each segmentation, and a preset path calculation weight algorithm; Step 606, dynamically planning the path required from the upper left corner to the lower right corner of the word segmentation mapping matrix using a path planning algorithm, and calculating the path weights corresponding to all paths based on the path calculation weights corresponding to each word in the first ordered word segmentation array and the second ordered word segmentation array; Step 607, selecting the minimum path weight from the path weights corresponding to all the paths, and calculating the repetition of the first text item and the second text item in combination with a preset repetition calculation formula; The step of performing text clustering processing on all the text items to be clustered according to the repetitiveness to obtain the text clustering result specifically includes: Identifying whether the repetitiveness exceeds a preset repetitiveness threshold; If it exceeds, classifying the second text entry into the text clustering set where the first text entry is located; If not, continue to select other text entries from the duplicate determination subset corresponding to the first text entry as the second text entry, and continue to execute steps 603 to 607.
7. The text clustering method according to claim 6, It is characterized in that The step of respectively calculating the path calculation weight corresponding to each segmentation in the first ordered segmentation array and the second ordered segmentation array according to the latest word frequency probability corresponding to each segmentation, the latest word weight corresponding to each segmentation, and a preset path calculation weight algorithm specifically includes: Use the preset path calculation weight algorithm: Calculate the path calculation weight algorithm corresponding to each word in the first ordered word segmentation array and the second ordered word segmentation array, where a represents the identification information of the current word segmentation, K a Indicates the word calculation weight of the current segmentation, P a Indicates the latest word frequency probability of the current word segmentation, and 0 <P a <1,ω a represents the word weight of the current word segmentation, and ω a >0, c represents the correction coefficient; The step of selecting the minimum path weight from the path weights corresponding to all the paths, and calculating the repetition of the first text item and the second text item in combination with a preset repetition calculation formula, specifically includes: The preset repeatability calculation formula is: Calculate the repetition degree of the first text entry and the second text entry, where m represents the total number of words in the ordered word segmentation array corresponding to the first text entry, n represents the total number of words in the ordered word segmentation array corresponding to the second text entry, l represents the minimum path weight, and min(m,n) represents selecting a relatively small value from m and n.
8. The text clustering method according to any one of claims 1 to 7, It is characterized in that The method further comprises: According to a preset time interval or the cumulative number of entries, counting the total number of occurrences of the specified words outside the common word dictionary in the target database, wherein the specified words include the latest words added by the operator through human-computer interaction; Determine whether the total number of times meets a preset number threshold; If the total number of times meets a preset number threshold, the designated word is entered into the common word dictionary as a new common word; If the total number of times does not meet the preset number threshold, there is no need to enter the designated word as a new common word into the common word dictionary.
9. A text clustering device, It is characterized in that include: A text acquisition and caching module, used to acquire the text set to be clustered and cache it in a target database, wherein a common word dictionary is preset in the target database, and the common word dictionary records the word frequency probability and word weight corresponding to all common words respectively; A calculation module, for calculating the latest word frequency probability and the latest word weight corresponding to all common words after the text set to be clustered is cached into the database based on the text entry changes of the target database before and after caching; An initial clustering identifier generating module, used for generating initial clustering identifiers for all the text items to be clustered in the text set to be clustered, wherein each text item to be clustered corresponds to a unique initial clustering identifier; A word segmentation processing module, used to perform word segmentation processing on all the text items to be clustered according to the common word dictionary to obtain word segmentation processing results; An initial duplicate detection module is used to sequentially filter out a duplicate detection subset corresponding to each to-be-clustered text entry from the to-be-clustered text set based on the word segmentation processing result, the initial clustering identifier and a preset preliminary duplicate detection strategy; A secondary duplication detection module is used to calculate the duplication degree of each text item to be clustered and each text item in its corresponding duplication detection subset by using the latest word frequency probability, the latest word weight and a preset secondary duplication detection strategy; The text clustering module is used to perform text clustering processing on all the text items to be clustered according to the repetition degree to obtain a text clustering result.
10. A computer device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the text clustering method according to any one of claims 1 to 8 when executing the computer-readable instructions.
11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the text clustering method according to any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Improved density text clustering algorithm
CN106528621A
A text feature selection method based on full-coverage granular computing
CN109165290A
System for generating feature vectors for long texts to realize classification
CN110413777A
Determining semantic similarity of texts based on sub-sections thereof
US20200302018A1
