Text deduplication method, apparatus, electronic device, and computer program product

Abstract

This application provides a method, apparatus, device, and program product for text deduplication, relating to the field of big data processing technology. The text deduplication method includes: mapping keywords of several texts to be deduplicated to obtain keyword vector sequences of several texts to be deduplicated; based on the keyword vector sequences of the texts to be deduplicated, using a rotation-twisting distance lower bound algorithm to filter several texts to be deduplicated that meet a first preset condition; based on the keyword vector sequences of the texts to be deduplicated that meet the first preset condition, using a rotation-twisting distance algorithm to obtain the similarity between the several texts to be deduplicated that meet the first preset condition; and performing deduplication based on the similarity level of the texts to be deduplicated that meet the first preset condition. The text deduplication method provided in this application, combining coarse filtering and precise deduplication, can first filter out obviously dissimilar texts to be deduplicated, reducing processing costs from the source, reducing technical difficulty for subsequent steps, and effectively improving deduplication efficiency.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, specifically to methods, apparatus, electronic devices, and computer program products for text deduplication. Background Technology

[0002] With the rapid development of internet technology, the speed and breadth of information dissemination have increased dramatically, greatly increasing the probability of collecting duplicate information. Therefore, it is necessary to deduplicate the collected text information to remove duplicates.

[0003] Existing text deduplication techniques typically use natural language processing models to compare the text to be deduplicated word by word to obtain accurate comparison results. Then, based on the comparison results, texts with high similarity are deduplicated. However, when the number of texts to be deduplicated is large, this method requires multiple processing threads and corresponding hardware resources, which increases costs and consumes more time, resulting in extremely low deduplication efficiency. Summary of the Invention

[0004] This application provides a text deduplication method to address the technical problem that when the number of texts to be deduplicated is large, the existing technology requires adding multiple processing threads and corresponding hardware resources for deduplication, which increases costs and consumes more time, resulting in extremely low deduplication efficiency.

[0005] In a first aspect, embodiments of this application provide a method for deduplicating text, including:

[0006] The keywords of several texts to be deduplicated are mapped to obtain several keyword vector sequences of the texts to be deduplicated;

[0007] Based on the keyword vector sequences of several texts to be deduplicated, a number of texts that meet the first preset conditions are selected using the rotation and twisting distance lower bound algorithm.

[0008] Based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, the similarity between several texts to be deduplicated that meet the first preset conditions is obtained using the rotation-twisting distance algorithm. Based on the similarity level, the overlapping and deduplication of several texts to be deduplicated that meet the first preset conditions is achieved.

[0009] In one embodiment, the step of filtering out several texts that meet a first preset condition based on the keyword vector sequences of several texts to be deduplicated using a rotation-twisting distance lower bound algorithm includes:

[0010] Based on the keyword vector sequences of several texts to be deduplicated, the lower bound of the word editing distance between several texts to be deduplicated is obtained by using the rotation and twisting distance lower bound algorithm;

[0011] The lower bound of the word editing distance is compared with the first preset condition to filter out several texts that meet the first preset condition for deduplication.

[0012] In one embodiment, the step of obtaining the lower bound of the word editing distance between the several texts to be deduplicated based on the keyword vector sequence of the several texts to be deduplicated is specifically as follows:

[0013] By combining the keyword vector sequences of several texts to be deduplicated and the magnitude of the keyword vector sequences of several texts to be deduplicated, the lower bound of the twisted word distance is obtained by using the lower bound expression through the rotation twisting distance lower bound algorithm.

[0014] In one embodiment, the lower bound expression is derived by combining the word-inducing machine distance algorithm and the rotation-twisting word distance algorithm, and the lower bound expression is:

[0015]

[0016] Among them, w i Let w' represent the sequence of keyword vectors for the i-th text to be deduplicated. j Let γ represent the sequence of keyword vectors for the j-th text to be deduplicated. ij Z represents the lower bound of the word edit distance, and Z represents w. i The normalized modulus length, Z' represents w' j The normalized modulus.

[0017] In one embodiment, the step of obtaining the similarity between the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions using a rotation-twisting distance algorithm, and then performing overlap deduplication on the several texts to be deduplicated that meet the first preset conditions based on the similarity scores, includes:

[0018] Based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, the word twisting distance between several texts to be deduplicated that meet the first preset conditions is obtained by the word twisting distance algorithm, and the similarity between several texts to be deduplicated that meet the first preset conditions is obtained according to the word twisting distance.

[0019] Arrange several texts that meet the first preset condition into an inverted triangle according to their similarity, thereby achieving the deduplication of the texts that meet the first preset condition.

[0020] In one embodiment, the keywords of several texts to be deduplicated are mapped to obtain several keyword vector sequences of the texts to be deduplicated. Specifically, the locality-sensitive hashing algorithm is used to map the keywords of several texts to be deduplicated into the keyword vector sequences of the texts to be deduplicated.

[0021] In one embodiment, the text deduplication method further includes: extracting keywords from several of the texts to be deduplicated using a term frequency-inverse text frequency index algorithm.

[0022] Secondly, embodiments of this application provide a text deduplication device, comprising:

[0023] The keyword mapping module is used to map the keywords of several texts to be deduplicated to obtain a sequence of keyword vectors for several texts to be deduplicated.

[0024] The deduplication text filtering module is used to filter out several deduplication texts that meet the first preset conditions based on the keyword vector sequence of several deduplication texts and using the rotation and twisting distance lower bound algorithm.

[0025] The text deduplication module is used to obtain the similarity between several texts that meet the first preset conditions based on the keyword vector sequences of several texts that meet the first preset conditions using a rotation and twisting distance algorithm, and to perform deduplication of several texts that meet the first preset conditions based on the similarity.

[0026] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the text deduplication method described in the first aspect.

[0027] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the text deduplication method described in the first aspect.

[0028] The text deduplication method, apparatus, electronic device, and computer program product provided in this application embodiment have no limit on the length of the text to be deduplicated. First, a rotation-twisting distance lower bound algorithm is used to roughly screen several texts to be deduplicated to identify those that meet a first preset condition. Then, a rotation-twisting distance algorithm is used to perform precise similarity calculation on the texts that meet the first preset condition, followed by duplicate deduplication processing. The text deduplication method, apparatus, electronic device, and computer program product provided in this application embodiment, by combining rough screening and precise deduplication, can first screen out obviously dissimilar texts to be deduplicated, reducing processing costs from the source, lowering the technical difficulty for subsequent processing steps, and effectively improving deduplication efficiency. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is one of the flowcharts illustrating the text deduplication method provided in the embodiments of this application;

[0031] Figure 2 This is the second flowchart illustrating the text deduplication method provided in the embodiments of this application;

[0032] Figure 3 This is a schematic diagram of the text deduplication device provided in the embodiments of this application;

[0033] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] Figure 1 This is one of the flowcharts for text deduplication methods.

[0036] Reference Figure 1 This application provides a method for deduplicating text, which may include:

[0037] S110. Map the keywords of several texts to be deduplicated to obtain a sequence of keyword vectors for several texts to be deduplicated.

[0038] S120. Based on the keyword vector sequence of several texts to be deduplicated, a number of texts to be deduplicated that meet the first preset condition are selected using the rotation and twisting distance lower bound algorithm.

[0039] S130. Based on the keyword vector sequence of several texts to be deduplicated that meet the first preset conditions, the similarity between several texts to be deduplicated that meet the first preset conditions is obtained by using the rotation and twisting distance algorithm, and the overlap and deduplication of several texts to be deduplicated that meet the first preset conditions is realized according to the similarity level.

[0040] It should be noted that the text deduplication method provided by this invention can be executed by any hardware device capable of text deduplication, such as a computer.

[0041] like Figure 2 As shown, before the hardware device capable of text deduplication executes S110, the hardware device capable of text deduplication will execute S100:

[0042] Collect texts to be deduplicated, and extract keywords from several of these texts.

[0043] In one embodiment, collecting the text to be deduplicated can, for example, collect news text information from major news websites; extracting keywords from the text to be deduplicated can be done using the Term Frequency-Inverse Text Frequency Index (TF-IDF) algorithm. Specifically, the hardware device capable of text deduplication can use the jieba word segmentation library based on the TF-IDF algorithm to extract keywords from the text to be deduplicated, using stop words, and determine the topk keyword value to be 20.

[0044] In S110, the hardware device capable of text deduplication maps the keywords of several texts to be deduplicated to obtain a sequence of keyword vectors for several texts to be deduplicated.

[0045] For example, hardware devices capable of text deduplication can use the Word2Vec model or the BERT model to map the keywords of the text to be deduplicated, thus obtaining a sequence of keyword vectors for the text to be deduplicated.

[0046] In S120, the hardware device capable of text deduplication will use a rotation-twisting distance lower bound algorithm to filter out several texts that meet the first preset conditions based on the keyword vector sequence of several texts to be deduplicated.

[0047] It should be noted that the rotation-twisting distance lower bound algorithm calculates the minimum degree of mismatch between the texts to be deduplicated. For example, the first preset condition can be a certain preset threshold. When the minimum degree of mismatch between the texts to be deduplicated obtained by the rotation-twisting distance lower bound algorithm is greater than the preset threshold, it indicates that the texts to be deduplicated are obviously dissimilar. S120 repeats this process among several texts to be deduplicated to filter out the texts to be deduplicated that meet the first preset condition. This can effectively reduce the processing cost of subsequent processing steps and avoid unnecessary processing and waste of resources.

[0048] In S130, the hardware device capable of text deduplication will use a rotation-twisting distance algorithm to obtain the similarity between the several texts that meet the first preset conditions based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, and perform overlapping deduplication of the several texts to be deduplicated that meet the first preset conditions according to the level of similarity.

[0049] It should be noted that the rotation-twisting distance algorithm calculates the minimum distance (i.e., the twisted word editing distance) that makes the keyword vector sequence of one text to be deduplicated become the keyword vector sequence of another text to be deduplicated. S130 obtains the precise similarity between texts to be deduplicated that meet the first preset condition through the rotation-twisting distance algorithm calculated in this way.

[0050] The text deduplication method provided in this application embodiment has no limit on the length of the text to be deduplicated. It first uses a rotation-twisting distance lower bound algorithm to roughly screen several texts to be deduplicated, identifying those that meet a first preset condition. Then, it uses a rotation-twisting distance algorithm to perform precise similarity calculation on these texts, followed by duplicate deduplication processing. This text deduplication method, combining rough screening and precise deduplication, can first identify obviously dissimilar texts to be deduplicated, reducing processing costs at the source and lowering the technical difficulty for subsequent processing steps, effectively improving deduplication efficiency.

[0051] In one embodiment, S110 specifically uses the Locality Sensitive Hash (SimHash) algorithm to map the keywords of several texts to be deduplicated into a sequence of keyword vectors of the texts to be deduplicated.

[0052] The implementation of the Locality Sensitive Hashing (LSH) algorithm includes: obtaining the features of the keywords in the text to be deduplicated; hashing the keyword features to obtain the keyword feature hash values; weighting the keyword feature hash values; merging and reducing the dimensionality of the weighted keyword feature hash values ​​to obtain the keyword vector sequence of the text to be deduplicated.

[0053] Hardware devices capable of text deduplication utilize the Locality Sensitive Hashing (LSH) algorithm to perform a unified digital transformation of unstructured text to be deduplicated, obtaining a sequence of keyword vectors for the text to be deduplicated. This ensures the uniformity of the format of the data to be processed subsequently, reduces the complexity of the processing, and effectively improves efficiency. Moreover, the LSH algorithm is simpler and more efficient to use than the Word2Vec model or the BERT model.

[0054] In one embodiment, S120 includes:

[0055] Based on the keyword vector sequences of several texts to be deduplicated, the lower bound of the word editing distance between several texts to be deduplicated is obtained by using the rotation and twisting distance lower bound algorithm;

[0056] The lower bound of the word editing distance is compared with the first preset condition to filter out several texts that meet the first preset condition for deduplication.

[0057] In one embodiment, based on the keyword vector sequence of several texts to be deduplicated, the lower bound of the word editing distance between the several texts to be deduplicated is obtained using the rotation and twisting distance lower bound algorithm, specifically as follows:

[0058] By combining the keyword vector sequences of several texts to be deduplicated and the magnitude of the keyword vector sequences of several texts to be deduplicated, the lower bound of the twisted word distance is obtained by using the lower bound expression through the rotation twisting distance lower bound algorithm.

[0059] In one embodiment, the lower bound expression is derived by combining the Word Mover's Distance (WMD) algorithm and the Word Rotator's Distance (WRD) algorithm, and the lower bound expression is:

[0060]

[0061] Among them, w i Let w' represent the sequence of keyword vectors for the i-th text to be deduplicated. j Let γ represent the sequence of keyword vectors for the j-th text to be deduplicated. ij Z represents the lower bound of the word edit distance, and Z represents w. i The normalized modulus length, Z' represents w' j The normalized modulus is obtained from top to bottom by first inequality based on Jensen's inequality and second inequality based on triangle inequality.

[0062] For example, the derivation of the lower bound expression is as follows:

[0063] Let the keywords of a sentence in the first text to be deduplicated be s = (t1, t2, ..., tn). n The keyword of a sentence in the second text to be deduplicated is s'=(t'1,t'2,…,t'). n’ Mapping these elements yields the corresponding keyword vector sequences (w1, w2, ..., w...). n ) and (w'1,w'2,…,w' n’If we use Wasserstein distance (Earth Mover's Distance, EMD) to compare the similarity of these two keyword vector sequences, we need to know p. i q j d i,j Three quantities, let p i ≡1 / n,q j ≡1 / n',d i,j The degree of difference between two keyword vector sequences can be represented by the Euclidean distance ||w||. i -w' j The symbol ‖ indicates that the degree of difference between two keyword vector sequences can be expressed as:

[0064]

[0065] Among them, w i Let w' represent the sequence of keyword vectors for the i-th text to be deduplicated. j Let γ represent the sequence of keyword vectors for the j-th text to be deduplicated. ij : This represents the optimal solution that makes the keyword vector sequence of the i-th text to be deduplicated the same as the keyword vector sequence of the j-th text to be deduplicated. st (short for subject to) basically means constrained by, subject to, or susceptible to; it can also mean under the condition of.

[0066] To better understand the Wasserstein distance, assume that p1, p2, ..., p are distributed at positions i = 1, 2, ..., n. n For simplicity, let's assume the total amount of soil is 1, i.e., p1 + p2 + ... + p n =1, if the soil is to be pushed to position j = 1, 2, ..., n', the amount of soil at each position j = 1, 2, ..., n' is q1, q2, ..., q n’ The cost of pushing the soil from point i to point j is d. i,j The solution with the lowest cost and the corresponding lowest cost is the optimal solution, which is denoted by γ. i,j This means that in this plan, γ needs to be removed from position i. i,j The quantity of soil is pushed to point j.

[0067] The word guessing machine distance algorithm is based on Wasserstein distance, but Wasserstein distance has two drawbacks:

[0068] Wasserstein distance uses Euclidean distance as a measure of semantic difference, but experience with the Word2Vec model in existing technologies shows that cosine distance is often better than Euclidean distance when calculating word vector similarity.

[0069] Wasserstein distance is theoretically an unbounded quantity, which means that the similarity can not be intuitively perceived through word-inference distance algorithms, resulting in poor similarity results.

[0070] The rotation-twisting distance algorithm asserts that "the magnitude of a word vector is positively correlated with the importance of the word." Therefore, combined with the word-inferring distance algorithm, it can be considered that p in the Wasserstein distance... i ,q j In a sense, it also represents the importance of a certain word in the corresponding sentence, so we can assume:

[0071]

[0072]

[0073] Then d i,j Represented using cosine distance:

[0074]

[0075] Where, d i,j p represents the cost of moving "soil" from point i to point j. i This represents probability, which can also be understood as a distribution with p1, p2, ..., p n So much "earth", q i This represents the probability density, which can also be understood as the amount of "soil" at each location being q1, q2, ..., q. n’ Z represents w i The normalized modulus length, Z' represents w' j The normalized modulus.

[0076] Substituting equations (2) and (3) into equation (1), we get:

[0077]

[0078] w i Let w' represent the sequence of keyword vectors for the i-th text to be deduplicated. j Let γ represent the sequence of keyword vectors for the j-th text to be deduplicated. ij : represents the optimal solution that makes the keyword vector sequence of the i-th text to be deduplicated the same as the keyword vector sequence of the j-th text to be deduplicated, and Z represents w i The normalized modulus length, Z' represents w' j The normalized modulus.

[0079] Then, based on equation (4), we can deduce that:

[0080]

[0081] Equation (5) is the lower bound expression, where w i Let w' represent the sequence of keyword vectors for the i-th text to be deduplicated. j Let γ represent the sequence of keyword vectors for the j-th text to be deduplicated. ij Z represents the lower bound of the word edit distance, and Z represents w. i The normalized modulus length, Z' represents w' j The normalized modulus is obtained from top to bottom by first inequality based on Jensen's inequality (or a generalized version of the basic inequality), and second inequality based on triangle inequality.

[0082] In one embodiment, S130 includes:

[0083] Based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, the word twisting distance between several texts to be deduplicated that meet the first preset conditions is obtained by the word twisting distance algorithm, and the similarity between several texts to be deduplicated that meet the first preset conditions is obtained according to the word twisting distance.

[0084] Arrange several texts that meet the first preset condition into an inverted triangle according to their similarity, thereby achieving the deduplication of the texts that meet the first preset condition.

[0085] When the number of texts to be deduplicated is large, forming an inverted triangle sort can more efficiently and accurately compare the texts to be deduplicated, so as to achieve the deduplication of overlapping texts.

[0086] In contrast, collecting 100 text entries to be deduplicated, with 4 million data entries to be matched, takes approximately 0.023 seconds per calculation. Directly performing precise deduplication requires (100 * 4 million) calculations, followed by sorting to obtain the results, with a minimum time consumption of 400 million * 0.023s = 106 days. The time to obtain a result for a single keyword vector sequence is 2.4 hours. However, when using the text deduplication method provided in this embodiment (combining coarse filtering and precise deduplication), setting an appropriate filter value, such as 0.88, only 126 seconds are needed to reduce the 4 million data entries to the 5K level. The time to obtain a result for a single keyword vector sequence is 0.023 * 5000 + 126 = 241 seconds. In practice, optimization measures such as multi-processing, multi-threading, or pagination can be used to further optimize this 241 seconds to within 10 seconds. Furthermore, the deduplication accuracy of the text deduplication method provided in this embodiment is over 99%.

[0087] The text deduplication method provided in this invention directly compares the differences between the variable-length sequences of the text to be deduplicated, rather than first converting the keyword vector sequence into a single vector for comparison. Therefore, there is no limitation on the length of the text to be deduplicated. This method can perform fast and accurate deduplication even with large datasets that may reach hundreds of millions or even billions of records and have varying text lengths.

[0088] The text deduplication apparatus provided in the embodiments of this application will be described below. The text deduplication apparatus described below can be referred to in correspondence with the text deduplication method described above.

[0089] A text deduplication device, such as Figure 3 As shown, it includes:

[0090] The keyword mapping module 310 is used to map the keywords of several texts to be deduplicated to obtain a sequence of keyword vectors of several texts to be deduplicated.

[0091] The text filtering module 320 is used to filter out a number of texts that meet the first preset conditions based on the keyword vector sequence of a number of texts to be deduplicated and using the rotation and twisting distance lower bound algorithm.

[0092] The text deduplication module 330 is used to obtain the similarity between the texts that meet the first preset conditions based on the keyword vector sequences of the texts that meet the first preset conditions using a rotation and twisting distance algorithm, and to perform deduplication of the texts that meet the first preset conditions based on the similarity.

[0093] In one embodiment, the text filtering module 320 for deduplication includes:

[0094] The word-twisting edit distance lower bound submodule is used to obtain the word-twisting edit distance lower bound between several texts to be deduplicated based on the keyword vector sequence of several texts to be deduplicated, using the word-twisting edit distance lower bound algorithm;

[0095] The word-twisting editing distance lower bound comparison submodule is used to compare the word-twisting editing distance lower bound with the first preset condition and filter out a number of texts to be deduplicated that meet the first preset condition.

[0096] In one embodiment, the submodule for obtaining the lower bound of the twisted word edit distance is specifically used to: combine the keyword vector sequences of several texts to be deduplicated and the magnitude of the keyword vector sequences of several texts to be deduplicated, and obtain the lower bound of the twisted word edit distance using the lower bound expression through the rotation twisted word distance lower bound algorithm.

[0097] In one embodiment, the lower bound expression of the word twisting editing distance lower bound submodule is derived by combining the word pusher distance algorithm and the rotation twisting distance algorithm, and the lower bound expression is:

[0098]

[0099] Among them, w i Let w' represent the sequence of keyword vectors for the i-th text to be deduplicated. j Let γ represent the sequence of keyword vectors for the j-th text to be deduplicated. ij Z represents the lower bound of the word edit distance, and Z represents w. i The normalized modulus length, Z' represents w' j The normalized modulus.

[0100] In one embodiment, the text overlap deduplication module 330 includes:

[0101] The word twisting edit distance submodule is used to obtain the word twisting edit distance between several texts to be deduplicated that meet the first preset conditions based on the keyword vector sequence of several texts to be deduplicated that meet the first preset conditions through the word twisting edit distance algorithm, and to obtain the similarity between several texts to be deduplicated that meet the first preset conditions based on the word twisting edit distance.

[0102] The text to be deduplicated arrangement submodule is used to arrange several texts to be deduplicated that meet the first preset condition into an inverted triangle shape according to their similarity, thereby realizing the deduplication of several texts to be deduplicated that meet the first preset condition.

[0103] In one embodiment, the keyword mapping module 310 is specifically used to map the keywords of several texts to be deduplicated into a keyword vector sequence of the texts to be deduplicated using a locality-sensitive hashing algorithm.

[0104] In one embodiment, the text deduplication device further includes:

[0105] The keyword extraction module is used to extract keywords from several texts to be deduplicated using the word frequency-inverse text frequency index algorithm.

[0106] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call a computer program in the memory 830 to execute the steps of a text deduplication method, such as including:

[0107] The keywords of several texts to be deduplicated are mapped to obtain several keyword vector sequences of the texts to be deduplicated;

[0108] Based on the keyword vector sequences of several texts to be deduplicated, a number of texts that meet the first preset conditions are selected using the rotation and twisting distance lower bound algorithm.

[0109] Based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, the similarity between several texts to be deduplicated that meet the first preset conditions is obtained using the rotation-twisting distance algorithm. Based on the similarity level, the overlapping and deduplication of several texts to be deduplicated that meet the first preset conditions is achieved.

[0110] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the text deduplication method provided in the above embodiments, such as including:

[0112] The keywords of several texts to be deduplicated are mapped to obtain several keyword vector sequences of the texts to be deduplicated;

[0113] Based on the keyword vector sequences of several texts to be deduplicated, a number of texts that meet the first preset conditions are selected using the rotation and twisting distance lower bound algorithm.

[0114] Based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, the similarity between several texts to be deduplicated that meet the first preset conditions is obtained using the rotation-twisting distance algorithm. Based on the similarity level, the overlapping and deduplication of several texts to be deduplicated that meet the first preset conditions is achieved.

[0115] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including:

[0116] The keywords of several texts to be deduplicated are mapped to obtain several keyword vector sequences of the texts to be deduplicated;

[0117] Based on the keyword vector sequences of several texts to be deduplicated, a number of texts that meet the first preset conditions are selected using the rotation and twisting distance lower bound algorithm.

[0118] Based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, the similarity between several texts to be deduplicated that meet the first preset conditions is obtained using the rotation-twisting distance algorithm. Based on the similarity level, the overlapping and deduplication of several texts to be deduplicated that meet the first preset conditions is achieved.

[0119] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0120] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for deduplicating text, characterized in that, include: The keywords of several texts to be deduplicated are mapped to obtain several keyword vector sequences of the texts to be deduplicated; Based on the keyword vector sequences of several texts to be deduplicated, a number of texts that meet the first preset conditions are selected using the rotation and twisting distance lower bound algorithm. Based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, the similarity between several texts to be deduplicated that meet the first preset conditions is obtained by using the rotation and twisting distance algorithm, and the overlap and deduplication of several texts to be deduplicated that meet the first preset conditions is realized according to the similarity level. The method involves using a keyword vector sequence of several texts to be deduplicated, and employing a rotation-twisting distance lower bound algorithm to select several texts that meet the first preset condition, including: Based on the keyword vector sequences of several texts to be deduplicated, the lower bound of the word editing distance between several texts to be deduplicated is obtained by using the rotation and twisting distance lower bound algorithm; The lower bound of the word twisting editing distance is compared with the first preset condition to filter out several texts that meet the first preset condition for deduplication. The method of obtaining the lower bound of the word editing distance between the several texts to be deduplicated based on the keyword vector sequence of several texts to be deduplicated is specifically as follows: By combining the keyword vector sequences of several texts to be deduplicated and the magnitude of the keyword vector sequences of several texts to be deduplicated, the lower bound of the twisted word distance is obtained by using the lower bound expression through the rotation twisting word distance lower bound algorithm; The lower bound expression is derived by combining the word-exploration machine distance algorithm and the rotation-twisting word distance algorithm. The lower bound expression is: ， Among them, w i Let w' represent the sequence of keyword vectors for the i-th text to be deduplicated. j Let γ represent the sequence of keyword vectors for the j-th text to be deduplicated. ij Z represents the lower bound of the word edit distance, and Z represents w. i The normalized modulus length, Z' represents w' j The normalized modulus.

2. The text deduplication method according to claim 1, characterized in that, The step involves using keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, employing a rotation-twisting distance algorithm to obtain the similarity between these texts, and then performing deduplication based on the similarity levels. This includes: Based on the keyword vector sequences of several texts to be deduplicated that meet the first preset conditions, the word twisting distance between several texts to be deduplicated that meet the first preset conditions is obtained by the word twisting distance algorithm, and the similarity between several texts to be deduplicated that meet the first preset conditions is obtained according to the word twisting distance. Arrange several texts that meet the first preset condition into an inverted triangle according to their similarity, thereby achieving the deduplication of the texts that meet the first preset condition.

3. The text deduplication method according to claim 1, characterized in that, The process involves mapping the keywords of several texts to be deduplicated to obtain a sequence of keyword vectors for those texts. Specifically, a Locality Sensitive Hashing (LSH) algorithm is used to map the keywords of the several texts to be deduplicated into a sequence of keyword vectors for those texts.

4. The text deduplication method according to claim 1, characterized in that, The text deduplication method further includes: extracting keywords from several of the texts to be deduplicated using a term frequency-inverse text frequency index algorithm.

5. A text deduplication device, characterized in that, The text deduplication device is used to implement the text deduplication method as described in any one of claims 1 to 4, comprising: The keyword mapping module is used to map the keywords of several texts to be deduplicated to obtain a sequence of keyword vectors for several texts to be deduplicated. The deduplication text filtering module is used to filter out several deduplication texts that meet the first preset conditions based on the keyword vector sequence of several deduplication texts and using the rotation and twisting distance lower bound algorithm. The text deduplication module is used to obtain the similarity between several texts that meet the first preset conditions based on the keyword vector sequences of several texts that meet the first preset conditions using a rotation and twisting distance algorithm, and to perform deduplication of several texts that meet the first preset conditions based on the similarity.

6. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the text deduplication method according to any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the text deduplication method according to any one of claims 1 to 4.

Images