Text deduplication method and device

Abstract

The present invention relates to the field of machine learning technology, and specifically provides a text deduplication method, characterized by comprising the following steps: S1, text standardization processing, performing relevant standardization processing on the acquired original text; S2, text word segmentation processing, developing a corresponding script program in Python language to perform word segmentation processing on the standardized text data; S3, text sentence segmentation processing; S4, setting a threshold; S5, calculating text feature values; and S6, outputting the results. Compared with the prior art, the present invention uses an efficient algorithm design to label similarity ratios between text data, and then, based on the set similarity threshold, screens out text data with excessive similarity for users to further dedupe or perform text relevance analysis. This reduces the homogeneity of network content, reduces the database storage burden, improves the quality of text content and the efficiency of information dissemination, and saves manpower and material costs.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and specifically provides a text deduplication method and device. Background Art

[0002] A data mining algorithm is a set of heuristics and calculations that creates a data mining model from data. To create the model, the algorithm first analyzes the provided data and looks for specific types of patterns and trends.

[0003] Statistical analysis is the process of analyzing large amounts of collected data using appropriate statistical methods to extract useful information and form conclusions, thereby conducting detailed research and summarizing the data. Statistical analysis of data leads to conclusions that can help people make decisions and take appropriate actions.

[0004] Data processing involves the collection, storage, retrieval, processing, transformation, and transmission of data. It extracts valuable and meaningful data from large amounts of potentially disorganized and incomprehensible data, and derives it from these data to create meaningful, meaningful data for specific individuals. Data processing is a fundamental component of systems engineering and automatic control, permeating all areas of social production and life, and significantly impacting the development of human society.

[0005] The rapid development and advancement of information technology has led to an accelerated rate of information dissemination and an unprecedentedly large amount of information generated. Within this vast amount of information, text information is particularly prominent. However, in many cases, this text contains a large amount of duplicate text, such as reprinted or plagiarized news, mass messages, and overwhelming advertising copy. This duplicate text not only contributes to the homogeneity of online content but also significantly increases the storage burden on databases, reducing the quality of text content and the efficiency of information dissemination. Therefore, an algorithm and device are needed to verify the similarity and remove duplicates from this text content. Summary of the Invention

[0006] The present invention aims to address the deficiencies of the above-mentioned prior art and provides a highly practical text deduplication method.

[0007] A further technical task of the present invention is to provide a text deduplication device that is rationally designed, safe and applicable.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A text deduplication method comprises the following steps:

[0010] S1. Text standardization processing: performing relevant standardization processing on the obtained original text;

[0011] S2. Text segmentation processing: Develop corresponding script programs for word segmentation processing on the standardized text data using Python language;

[0012] S3, text sentence processing;

[0013] S4. Set the threshold;

[0014] S5, calculating text feature values;

[0015] S6. Output the results.

[0016] Furthermore, in step S1, the obtained original text is subjected to relevant standardization processing, including the following steps:

[0017] S201. Remove all visible non-Chinese characters including numbers, letters and punctuation marks in the text;

[0018] S202, removing all invisible characters such as spaces, tabs, and line breaks from the text;

[0019] S203: Perform half-width conversion on the text to ensure that the text data to be compared is in a unified mode.

[0020] Furthermore, in step S2, the standardized text data is segmented by developing a corresponding script program in Python language, and the length L of each text before segmentation, the length L1 of each word after segmentation, and the number of times N that it appears in the current text are counted;

[0021] Furthermore, in step S3, a corresponding script program is developed for sentence segmentation of the standardized text data using Python language, with a period as the separator between sentences, and the length L2 of each sentence is counted.

[0022] Furthermore, in step S4, the device threshold includes the following:

[0023] (1) The number of words extracted from the text segmentation n1;

[0024] (2) The number of sentences extracted from the text segmentation n2;

[0025] (3) Similarity lower limit ε.

[0026] Furthermore, in step S5, the text feature values are calculated. For any two texts, the following steps are performed in sequence:

[0027] (1) Calculate the ratio of each word to the text: R1 = L1*N / L;

[0028] (2) Calculate the ratio of each clause to the text: R2 = L2 / L;

[0029] (3) Arrange the words in descending order according to R1, and extract n1 words according to the set number of words to be extracted from the text;

[0030] (4) Arrange the sentences in descending order according to R2, and extract n2 sentences according to the set number of text sentence extraction;

[0031] (5) Compare the two groups of words and extract the number of identical words m1, with a proportion of Z1 = m1 / n1;

[0032] (6) Compare the two groups of sentences and extract the number of identical sentences m2, with a proportion of Z2 = m2 / n2;

[0033] (7) Calculate the similarity between two texts: X = (Z1 + Z2) / 2.

[0034] Furthermore, in step S6, the result is output. For any two texts, the similarity between the two texts is calculated to be X, and the following conclusions are drawn:

[0035] (1) When X is 0, the two texts are completely unrelated;

[0036] (2) The smaller X is, the lower the similarity between the two texts is and the smaller the correlation is;

[0037] (3) The larger X is, the higher the similarity between the two texts is and the greater the correlation is;

[0038] (4) If X>ε, the two texts are considered to have the same content and can be deduplicated.

[0039] A text deduplication device, comprising: at least one memory and at least one processor;

[0040] The at least one memory is configured to store a machine-readable program;

[0041] The at least one processor is configured to call the machine-readable program to execute a text deduplication method.

[0042] Compared with the prior art, the text deduplication method and device of the present invention have the following outstanding beneficial effects:

[0043] The present invention uses machine learning to perform in-depth analysis and processing of text data. Through efficient algorithm design, it labels the text data with similarity ratios. Then, based on the set similarity threshold, it filters out text data with excessively high similarity for users to further deduplicate or perform text relevance analysis. This will reduce the homogeneity of network content, alleviate the database storage burden, improve the quality of text content and the efficiency of information dissemination, and save manpower and material costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Attachment Figure 1 This is a flowchart of a text deduplication method. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0047] A best embodiment is given below:

[0048] like Figure 1 As shown, a text deduplication method in this embodiment has the following steps:

[0049] S1. Text standardization processing: performing relevant standardization processing on the obtained original text, including the following steps:

[0050] S201. Remove all visible non-Chinese characters including numbers, letters and punctuation marks in the text;

[0051] S202, removing all invisible characters such as spaces, tabs, and line breaks from the text;

[0052] S203: Perform half-width conversion on the text to ensure that the text data to be compared is in a unified mode.

[0053] S2. Text segmentation processing: For the standardized text data, develop a corresponding script program through Python language to perform word segmentation processing, and count the length L of each text before word segmentation, the length L1 of each word after word segmentation, and the number of times N that it appears in the current text.

[0054] S3, text sentence processing;

[0055] For the standardized text data, a corresponding script program is developed using Python language to perform sentence processing. The separator between sentences is a period, and the length L2 of each sentence is counted.

[0056] S4. Set the threshold, including the following:

[0057] (1) The number of words extracted from the text segmentation n1;

[0058] (2) The number of sentences extracted from the text segmentation n2;

[0059] (3) Similarity lower limit ε.

[0060] S5. Calculate the text feature value. For any two texts, perform the following steps in sequence:

[0061] (1) Calculate the ratio of each word to the text: R1 = L1*N / L;

[0062] (2) Calculate the ratio of each clause to the text: R2 = L2 / L;

[0063] (3) Arrange the words in descending order according to R1, and extract n1 words according to the set number of words to be extracted from the text;

[0064] (4) Arrange the sentences in descending order according to R2, and extract n2 sentences according to the set number of text sentence extraction;

[0065] (5) Compare the two groups of words and extract the number of identical words m1, with a proportion of Z1 = m1 / n1;

[0066] (6) Compare the two groups of sentences and extract the number of identical sentences m2, with a proportion of Z2 = m2 / n2;

[0067] (7) Calculate the similarity between two texts: X = (Z1 + Z2) / 2.

[0068] S6. Output the result. For any two texts, the similarity between the two texts is calculated to be X. The following conclusions can be drawn:

[0069] (1) When X is 0, the two texts are completely unrelated;

[0070] (2) The smaller X is, the lower the similarity between the two texts is and the smaller the correlation is;

[0071] (3) The larger X is, the higher the similarity between the two texts is and the greater the correlation is;

[0072] (4) If X>ε, the two texts are considered to have the same content and can be deduplicated.

[0073] Based on the above method, a text deduplication device in this embodiment includes: at least one memory and at least one processor;

[0074] The at least one memory is configured to store a machine-readable program;

[0075] The at least one processor is configured to call the machine-readable program to execute a text deduplication method.

[0076] The above-mentioned specific implementation methods are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementation methods. Any text deduplication method and device claims that comply with the present invention and any appropriate changes or substitutions made by ordinary technicians in the technical field should fall within the patent protection scope of the present invention.

[0077] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A text deduplication method, characterized in that: The steps are as follows: S1. Text standardization processing: performing relevant standardization processing on the obtained original text; The following steps are involved: S101. Remove all visible non-Chinese characters including numbers, letters and punctuation marks in the text; S102, removing all invisible characters such as spaces, tabs, and line breaks from the text; S103, performing half-width conversion on the text to ensure that the text data to be compared is in a unified mode; S2. Text segmentation processing: Develop corresponding script programs for word segmentation processing on the standardized text data using Python language; For the standardized text data, use Python language to develop a corresponding script program to perform word segmentation processing, and count the length L of each text before word segmentation, the length L1 of each word after word segmentation, and the number of times N that it appears in the current text; S3, text sentence processing; For the standardized text data, a corresponding script program is developed using Python language to perform sentence processing, with the sentence separator being a period, and the length L2 of each sentence is counted; S4. Set the threshold; These include: (1) The number of words extracted from the text segmentation n1; (2) The number of sentences extracted from the text segmentation n2; (3) Similarity lower limit ε; S5, calculating text feature values; For any two texts, perform the following steps in sequence: (1) Calculate the ratio of each word to the text: R1 = L1*N / L; (2) Calculate the ratio of each clause to the text: R2 = L2 / L; (3) Arrange the words in descending order according to R1, and extract n1 words according to the set number of words to be extracted from the text; (4) Arrange the sentences in descending order according to R2, and extract n2 sentences according to the set number of text sentence extraction; (5) Compare the two groups of words and extract the number of identical words m1, with a proportion of Z1 = m1 / n1; (6) Compare the two groups of sentences and extract the number of identical sentences m2, with a proportion of Z2 = m2 / n2; (7) Calculate the similarity between two texts: X = (Z1 + Z2) / 2; S6. Output the results.

2. A text deduplication method according to claim 1, characterized in that: In step S6, the result is output. For any two texts, the similarity between the two texts is calculated to be X. The following conclusions can be drawn: (1) When X is 0, the two texts are completely unrelated; (2) The smaller X is, the lower the similarity between the two texts is and the smaller the correlation is; (3) The larger X is, the higher the similarity between the two texts is and the greater the correlation is; (4) If X>ε, the two texts are considered to have the same content and can be deduplicated.

3. A text deduplication device, characterized in that: include: at least one memory and at least one processor; The at least one memory is configured to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 2.

Images