A method for real-time deduplication clustering of long texts

Abstract

The application discloses a method for real-time deduplication and clustering of long texts, comprising the following steps: step one, infrastructure construction: a data structure capable of quickly fuzzy comparison of feature vectors is established in a central service storage; a text list A with feature values as primary keys is established in a central database; a text list B with feature values as primary keys is established in the central database; step two, coarse classification, real-time calculation when each text in the text list A and the text list B enters the system; step three, timed background modeling; step four, fine classification, timed and quantitative calculation is performed on the text list A, and all similar text pairs thus judged are marked with the minimum feature value in the set as the associated feature value in the table A; and step five, real-time query of terminal users: single text query and time period query, and the application relates to the technical field of data analysis. According to the application, the real-time data processing is ensured, and high accuracy is also considered.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a method for real-time deduplication and clustering of long texts. Background Technology

[0002] In public opinion analysis, deduplication and clustering of the acquired text is the foundation for all further analysis. However, since the acquired text is usually mixed with a lot of irrelevant noise information, such as multiple advertisements, and the text itself may have been modified, such as changing the title or deleting some content, traditional direct string comparison is basically ineffective, and existing complex methods cannot meet the real-time data processing requirements of massive data. Therefore, we propose a method for real-time deduplication and clustering of long texts. Summary of the Invention

[0003] To address the problem that existing deduplication and clustering methods for acquiring text in public opinion analysis cannot meet the real-time data processing requirements of massive datasets, the purpose of this invention is to provide a real-time deduplication and clustering method for long texts.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a method for real-time deduplication and clustering of long texts, comprising the following steps:

[0005] Step 1, Infrastructure Construction: Establish a data structure in the central service storage that can perform fast fuzzy comparison of feature vectors; establish a text list A with feature values ​​as the primary key in the central database; establish a text list B with feature values ​​as the primary key in the central database;

[0006] Step 2, coarse classification: When each text in text list A and text list B enters the system, the following calculations are performed in real time: extract n-grams as feature values ​​from the text; repeatedly perform hash calculations on all feature values ​​of each text, and calculate the results to obtain a feature vector of equal length for each text; pass the feature values ​​to the fuzzy ratio data structure for calculation to obtain the maximum similarity value;

[0007] Step 3, Timed Background Modeling: Query text list A, read text list B, and perform modeling of the distributed representation computing model, the deep learning semantic vector computing model, and the positive and negative data vector judgment model;

[0008] Step 4: Refine the classification. Perform timed and quantitative calculations on text list A to obtain a similarity matrix. Judge the similarity matrix according to the set threshold, and mark the minimum feature value of the associated feature value in table A for all text pairs judged to be similar.

[0009] Step 5, real-time query for end users: single text query and time period query.

[0010] Preferably, in step one, text list A includes a counter, text content, acquisition time, and associated feature values; text list B includes text content and associated feature values.

[0011] Preferably, in step two, if the maximum similarity value is greater than the set highest threshold, then this text is considered duplicate data, and the counter attached to the corresponding primary key in text list A is incremented by 1; if the maximum similarity is not greater than the set highest threshold, then this text is considered new text, and the corresponding feature value and acquisition time are stored in text list A, with the counter set to 1; if the maximum similarity is not greater than the set highest threshold but greater than the set lowest threshold, then the text is stored in text list B, and the feature value of the text most similar to it is stored in text list A; if any method can determine the similarity of text pairs during the business process, then it is also added to text list B; if the maximum similarity is greater than the set highest threshold, then the coarse classification return value is the feature value of the text most similar to it, otherwise it is its own feature value.

[0012] Preferably, in step three, at a set time, the text list A is queried to obtain all texts within the set time period; all obtained texts are segmented and stop words are removed; unsupervised learning is performed on all segmented texts using a distributed representation method to obtain word frequencies and semantic vectors corresponding to each word; unsupervised learning is performed on the semantic vector stream of each text using a deep learning method to obtain the semantic vector of each text; the text list B is read to obtain similar texts as positive training data for supervised learning, while text pairs are randomly selected from text list A as negative training data for supervised learning; a linear model is trained so that the Cartesian product of the vectors of the positive data pairs obtained in the previous step is as large as possible after linear transformation, while the Cartesian product of the vectors of the negative data pairs is as orthogonal as possible after linear transformation, thereby obtaining the modeling of the distributed representation calculation model, the deep learning semantic vector calculation model, and the positive and negative data vector judgment model.

[0013] Preferably, in step four, at a set time, the text list A is queried to obtain the text within the set time period; the text is processed using a distributed representation computing model, a deep learning semantic vector computing model, and the linear part of the positive and negative data vector judgment model to obtain a tensor of the text set; the tensor of the text set is processed by itself to calculate the inner product to obtain a similarity matrix.

[0014] Preferably, in step five, a single text query involves performing a coarse classification calculation on the text, and then querying the obtained feature values ​​in text list A: a counter represents the number of texts with the same basic content as this text; by looking up the associated feature values ​​in the table, the difference between the text content and the queried content can be obtained.

[0015] Preferably, in step five, the time period query is performed by querying the time period recorded in Table A, and then classifying the relevant feature values ​​in the query results to obtain the text set for this time period. Each text set includes a text subset that describes the same event. The counter attached to the text in the subset is the number of other texts that are basically similar to the text.

[0016] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0017] 1. In this invention, all text deduplication and clustering queries can be performed in real time;

[0018] 2. This invention achieves a recognition rate of nearly 100% for similar texts that have been simply modified and contain varying degrees of noise.

[0019] 3. This invention has a high recognition rate for similar texts rewritten based on meaning, and the recognition rate will continue to improve as data is continuously collected; if a certain amount of manual annotation is added, the concept of "similar text" in the algorithm can also be adjusted.

[0020] 4. According to this invention, the system can run completely unsupervised, or a certain amount of manual annotation can be added to make it more in line with the business direction; in the case of complete unsupervised operation, the system model training and other processes are triggered automatically without manual maintenance.

[0021] 5. In this invention, each model training session will automatically have a positive impact on business queries. Therefore, the system can be adjusted by adjusting parameters to serve different projects with different requirements for accuracy and computational cost. Attached Figure Description

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0023] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0024] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0025] Please see Figure 1It should be understood that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are merely for illustrative purposes to aid those skilled in the art and to facilitate understanding and reading. They are not intended to limit the scope of the invention and therefore have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effectiveness and purpose of the invention, should still fall within the scope of the technical content disclosed in this invention. Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity and not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention's implementation.

[0026] This invention provides a technical solution:

[0027] I. Infrastructure Construction

[0028] a) Establish a data structure (LSHforest) in the central service storage that can perform fast fuzzy comparison of feature vectors;

[0029] b) Create a text list (Table A) in the central database with feature values ​​as the primary key. The table columns should include at least the counter, text content, acquisition time, and associated feature value.

[0030] c) At the same time, create a text list (Table B) in the central database with the feature value as the primary key. The table columns include the text content and the associated feature value.

[0031] II. Coarse Classification: Calculated in real time as each text enters the system.

[0032] a) No semantic processing such as word segmentation is performed on the text; instead, n-grams are directly extracted from the text as feature values.

[0033] b) Repeatedly perform hash calculations on all feature values ​​of each text, and calculate the results to obtain a feature vector (MinHash) of equal length for each text;

[0034] c) Pass the feature values ​​to the fuzzy ratio data structure for calculation to obtain the maximum similarity value;

[0035] d) If the maximum similarity value is greater than the set maximum threshold, then this text is duplicate data, and the counter attached to the corresponding primary key in the text list (Table A) is incremented by 1;

[0036] e) If the maximum similarity is not greater than the set maximum threshold, then this text is determined to be a (preliminary screening) new text, and the corresponding feature value and acquisition time are stored in the text list (Table A), and the counter is set to 1;

[0037] f) If the maximum similarity is not greater than the set highest threshold, but is greater than the set lowest threshold, then the text is stored in Table B, and the feature value of the text most similar to it is stored in Table A.

[0038] g) If there is any method in the business process to determine that text pairs are similar, then add them to Table B as well;

[0039] h) If the maximum similarity is greater than the set maximum threshold, the coarse classification returns the feature value of the text most similar to it; otherwise, it returns its own feature value.

[0040] III. Timed Background Modeling

[0041] a) At a set time, query text list A to get all text within the set time period (e.g., within one year);

[0042] b) Perform word segmentation and stop word removal on all acquired texts;

[0043] c) Use a distributed representation method to perform unsupervised learning on all the segmented text to obtain word frequencies and semantic vectors for each word;

[0044] d) Use deep learning methods to perform unsupervised learning on the semantic vector stream of each text to obtain the semantic vector of each text;

[0045] e) Read text list B, obtain similar texts as positive training data for supervised learning, and at the same time randomly select text pairs from text list A as negative training data for supervised learning.

[0046] f) Train a linear model such that the Cartesian product of the vectors of the positive data pairs obtained in the previous step is as large as possible after linear transformation, while the Cartesian product of the vectors of the negative data pairs is as orthogonal as possible after linear transformation.

[0047] g) Store the three models: the distributed representation computation model, the deep learning semantic vector computation model, and the positive / negative data vector judgment model, for use in the next step;

[0048] IV. Detailed Classification: Timed and Quantitative Calculation

[0049] a) At a set time, query text list A to retrieve texts within a set time period (the time period that the public opinion project focuses on, such as one week).

[0050] b) The text set tensor is obtained by using a distributed representation computing model, a deep learning semantic vector computing model, and the linear part of the positive and negative data vector judgment model.

[0051] c) Calculate the inner product of the tensors of the text set to obtain the similarity matrix;

[0052] d) Based on the set threshold, the similarity matrix is ​​judged, and all text pairs judged as similar are marked in Table A with the smallest feature value in this set as the associated feature value;

[0053] V. Real-time query by end users

[0054] a) Single text query: Perform coarse classification calculations on the text, and then query the obtained feature values ​​in Table A.

[0055] i. The counter represents the number of texts with the same basic content as this text.

[0056] ii. By performing a reverse lookup based on the associated feature values ​​in the table, we can obtain text whose content differs slightly from the queried content, but which describes the same basic event;

[0057] b) Time period query: Perform a time period query based on the time recorded in Table A, and then classify the relevant feature values ​​in the query results to obtain the text set for this time period. Each text set includes a text subset that describes the same event. The counter attached to the text in the subset is the number of other texts that are basically the same as the text.

[0058] Explanation of time period and resource consumption

[0059] Rough classification:

[0060] For real-time calculation, the computational cost per piece of content is very low, and all text in the output is consistent.

[0061] The rejection ratio varies depending on the text source and crawling strategy. Based on experience, coarse classification can reject 50% to 90% of the text as duplicates.

[0062] Detailed classification:

[0063] It requires relatively resource-intensive calculations such as word segmentation and vector calculation of the text, with a medium resource consumption.

[0064] Based on two main settings:

[0065] The update frequency is typically calculated every half day or every day, meaning that new subcategories are generated at this frequency.

[0066] Calculation period: Determined by the business, usually the most recent week. This value represents the business's main focus on the clustering and dissemination of new information within this period.

[0067] Background modeling:

[0068] Training deep models on massive amounts of data (hundreds of millions of records) requires GPU clusters for computation, resulting in high resource consumption.

[0069] The resulting model is mainly used for semantic modeling, so it has strong universality. Unless the overall style of the text changes significantly, it can be used indefinitely. This calculation is usually performed every few months (to prevent the machine from "not understanding" a large number of new online words generated within a few months).

[0070] Terminal query:

[0071] It only performs coarse classification and database table lookups, so it can respond in real time;

[0072] At the same time, due to the hierarchical association design, the latest incoming text can not only find texts that are similar to it, but also texts that describe the same content.

[0073] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for real-time deduplication clustering of long texts, characterized in that, Comprise the following steps: Step one, infrastructure construction: in the center service storage can be established on the feature vector fuzzy comparison of data structure fast; in the center database to establish the feature value as the primary key of the text list A, text list A includes counter, text content, access time, associated feature value; in the center database to establish the feature value as the primary key of the text list B, text list B includes text content and associated feature value; Step two, rough classification, in real time when each text in text list A and text list B enters the system: n-gram is extracted from the text as a feature value; Repeat the hash calculation of all feature values of each text, and calculate the calculation results to obtain the equal length feature vector of each text; The feature value is calculated with the fuzzy comparison data structure to obtain the maximum similarity value; If the maximum similarity value is greater than the highest threshold value, the text is repeated data, and the corresponding primary key in the text list A is increased by 1; If the maximum similarity is not greater than the highest threshold value, it is judged that the text is a new text, and the corresponding feature value, access time and counter are stored in the text list A; If the maximum similarity is not greater than the highest threshold value, but is greater than the lowest threshold value, the text is stored in the text list B, and the most similar text feature value is stored in the text list A; If there is any method to judge the similarity of the text in the business process, the text list B is also added; If the maximum similarity is greater than the highest threshold value, the return value of rough classification is the feature value of the most similar text, otherwise it is its own feature value; Step three, background modeling: query text list A, read text list B, and perform storage distributed representation calculation model, deep learning semantic vector calculation model, and positive and negative data vector judgment model modeling; At the set time, query text list A and obtain all texts in the set time period; perform word segmentation and stop word removal on all obtained texts; Use the distributed representation method to perform unsupervised learning on all segmented texts to obtain word frequency and semantic vector corresponding to each word; use deep learning method to perform unsupervised learning on the semantic vector stream of each text to obtain the semantic vector of each text; Read text list B, obtain similar texts as positive training data for supervised learning, and randomly extract texts from text list A as negative training data for supervised learning; Train a linear model so that the vector of the positive data pair obtained in the previous step is as large as possible after linear transformation, and the vector of the negative data pair is as orthogonal as possible after linear transformation, thereby obtaining the modeling of the storage distributed representation calculation model, the deep learning semantic vector calculation model and the positive and negative data vector judgment model. Step four, fine classification, perform timed and quantitative calculation on text list A, query text list A at the set time, and obtain texts in the set time period; The linear part of the distributed representation calculation model, the deep learning semantic vector calculation model, and the positive and negative data vector judgment model is used for the text to obtain the tensor of the text set; the tensor of the text set is subjected to inner product calculation to obtain a similarity matrix, the similarity matrix is judged according to a set threshold, and all the text pairs judged to be similar are marked with the minimum feature value in the set in table A; Step five, terminal user real-time query: single text query and time period query; Single text query, coarse classification calculation is performed on the text, and then the obtained feature value is queried in the text list A: the counter represents the number of texts basically same as the text; the text content is distinguished from the queried content according to the reverse query of the associated feature value in the table; Time period query: according to the time recorded in table A, the period is queried, and then the associated feature value in the query result is classified, so that the text set in the period is obtained, and each text set in the text set includes a text subset, which collectively describes the same event; The counter of the text in the subset is the number of other texts basically same as the text.

Images