Corpus similarity detection method, device, intelligent terminal and storage medium

Abstract

The present invention discloses a corpus similarity detection method, device, intelligent terminal and storage medium, wherein the corpus similarity detection method comprises: obtaining an input document set; performing distributed preprocessing on each input document in the input document set to obtain a set of coded documents to be detected, wherein the distributed preprocessing includes character removal, word segmentation and word format conversion, the set of coded documents to be detected includes multiple coded documents to be detected, and each word in the coded document to be detected is an integer type code; corpus similarity detection is performed based on the set of coded documents to be detected. Compared with the prior art, in the present invention, when comparing documents, only the words in the documents need to be compared respectively, and only integer type data comparison needs to be performed, which is conducive to reducing the amount of calculation, reducing the comparison time, and improving the efficiency of corpus similarity detection.

Description

Technical Field

[0001] The present invention relates to the technical field of data similarity comparison, and in particular to a corpus similarity detection method, device, intelligent terminal and storage medium. Background Art

[0002] With the development of science and technology, various large-scale models are gradually increasing. Large-scale pre-trained models require a large amount of high-quality text corpus data as training samples. At present, most of the text corpus data used for training are crawled or collected from various data sources. There is a lot of redundant information in the collected data, and corpus similarity detection is needed to eliminate the corresponding redundant information.

[0003] In the prior art, all documents are usually directly compared pairwise, which has a large amount of calculation (equivalent to the Cartesian product of the two), and the pairwise string comparison of long documents itself has a large amount of calculation and takes a long time. The problem of the prior art is that the solution of directly comparing all documents pairwise to realize corpus similarity detection has a large amount of calculation and takes a long time, which is not conducive to improving the efficiency of corpus similarity detection.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a corpus similarity detection method, device, intelligent terminal and storage medium, aiming to solve the problem in the prior art that the solution of directly comparing all documents pairwise to achieve corpus similarity detection has large computational complexity, is time-consuming, and is not conducive to improving the efficiency of corpus similarity detection.

[0006] In order to achieve the above object, the present invention provides a corpus similarity detection method in a first aspect, wherein the method comprises:

[0007] Get the input document collection;

[0008] Performing distributed preprocessing on each input document in the input document set to obtain a set of coded documents to be detected, wherein the distributed preprocessing includes character removal, word segmentation and word format conversion, the set of coded documents to be detected includes multiple coded documents to be detected, and each word in the coded documents to be detected is an integer type code;

[0009] Corpus similarity detection is performed based on the above-mentioned set of coded documents to be detected.

[0010] Optionally, the above input document is a Chinese document.

[0011] Optionally, the input document collection is a resilient distributed dataset, and the distributed preprocessing and the corpus similarity detection are performed based on a Spark distributed framework.

[0012] Optionally, the distributed preprocessing further includes global numbering, stop word removal and precise deduplication. The distributed preprocessing is performed on each input document in the input document set to obtain a set of coded documents to be detected, including:

[0013] Globally numbering each of the input documents in the input document set to obtain a numbered input document set;

[0014] Respectively remove characters other than the target character from each numbered input document to obtain each target numbered document, wherein the target character is a Chinese character;

[0015] Perform word segmentation on each of the above target numbered documents respectively to obtain the target word array corresponding to each target numbered document;

[0016] Broadcasting the stop word list to each computing node, and removing the stop words in each of the target word arrays based on the stop word list at each of the computing nodes to obtain a word array to be detected;

[0017] Accurately remove duplicates from the set consisting of all the above-mentioned word arrays to be detected to obtain a document set to be detected, wherein the above-mentioned document set to be detected includes multiple target detection word arrays;

[0018] Each word in each of the above target detection word arrays is converted into an integer code respectively, each target detection code array is obtained, and a set consisting of all target detection code arrays is used as the above code document set to be detected.

[0019] Optionally, the corpus similarity detection based on the above-mentioned set of coded documents to be detected includes:

[0020] Construct a target word frequency table based on the above-mentioned set of coded documents to be detected;

[0021] Based on the target word frequency table, similarity detection is performed on the above-mentioned set of coded documents to be detected.

[0022] Optionally, the above-mentioned construction of a target word frequency table based on the above-mentioned set of coded documents to be detected includes:

[0023] Expand the word set of the above-mentioned coded document set to be detected to obtain an initial word frequency table, wherein the above-mentioned initial word frequency table includes A tuples, each tuple includes a word and the word frequency corresponding to the word, the above-mentioned A is the number of words in the above-mentioned coded document set to be detected, and each word frequency in the above-mentioned initial word frequency table is 1;

[0024] Merge the tuples with the same word code in the above initial word frequency table, and sum up the corresponding word frequencies to obtain a complete word frequency table;

[0025] The target word frequency table is obtained after deleting the low-frequency tuples in the above complete word frequency table, and the word frequency of the above low-frequency tuples is less than or equal to the preset word frequency threshold.

[0026] Optionally, the above preset word frequency threshold is 1.

[0027] Optionally, the similarity detection on the set of coded documents to be detected based on the target word frequency table includes:

[0028] In the above-mentioned set of code documents to be detected, the words in each of the above-mentioned target detection code arrays are sorted according to the word frequency in the above-mentioned target word frequency table to obtain each sorted code array;

[0029] Obtain the prefix word arrays corresponding to the above sorted encoding arrays respectively;

[0030] Partitioning and grouping the target detection encoding array based on the prefix word array;

[0031] Similarity verification is performed based on the partitioned and grouped target detection encoding array.

[0032] Optionally, partitioning and grouping the target detection encoding array based on the prefix word array includes: dividing the target detection encoding arrays having at least one common word in the prefix into the same group of the same partition of Spark.

[0033] Optionally, the similarity verification based on the partitioned and grouped target detection encoding arrays may include:

[0034] Based on a preset similarity calculation formula, similarity verification is performed on the target detection code array in each group, and the document number corresponding to the similar detection code array is obtained, wherein the similar detection code array is a target detection code array that meets the preset similarity condition;

[0035] Based on the above document number, the corresponding input document that meets the preset similarity condition is obtained.

[0036] A second aspect of the present invention provides a corpus similarity detection device, wherein the device comprises:

[0037] An input document set acquisition module is used to acquire an input document set;

[0038] A distributed preprocessing module is used to perform distributed preprocessing on each input document in the above-mentioned input document set to obtain a set of coded documents to be detected, wherein the above-mentioned distributed preprocessing includes character removal, word segmentation and word format conversion, the above-mentioned set of coded documents to be detected includes multiple coded documents to be detected, and each word in the above-mentioned coded documents to be detected is an integer type code;

[0039] The corpus similarity detection module is used to perform corpus similarity detection based on the above-mentioned coded document set to be detected.

[0040] A third aspect of the present invention provides an intelligent terminal, which includes a memory, a processor, and a corpus similarity detection program stored in the memory and executable on the processor. When the corpus similarity detection program is executed by the processor, the steps of any one of the corpus similarity detection methods are implemented.

[0041] A fourth aspect of the present invention provides a computer-readable storage medium, on which a corpus similarity detection program is stored. When the corpus similarity detection program is executed by a processor, the steps of any one of the corpus similarity detection methods are implemented.

[0042] As can be seen from the above, in the scheme of the present invention, an input document set is obtained; each input document in the above input document set is subjected to distributed preprocessing to obtain a set of coded documents to be detected, wherein the above distributed preprocessing includes character removal, word segmentation and word format conversion, the above coded document set to be detected includes multiple coded documents to be detected, and each word in the above coded document to be detected is an integer type code; corpus similarity detection is performed based on the above coded document set to be detected. Compared with the scheme of directly comparing documents by two strings in the prior art, the present invention first performs distributed preprocessing such as character removal, word segmentation, and word format conversion on the input document to obtain a set of coded documents to be detected. Specifically, each coded document to be detected includes words after word segmentation, and each word has been converted into an integer type code. In this way, when comparing documents, only the words in the documents need to be compared separately, and only integer type data comparison needs to be performed, which is conducive to reducing the amount of calculation, reducing the comparison time, and improving the efficiency of corpus similarity detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 It is a flowchart of a corpus similarity detection method provided by an embodiment of the present invention;

[0045] Figure 2 Embodiment of the present invention Figure 1 Specific flow diagram of step S200;

[0046] Figure 3Embodiment of the present invention Figure 1 Specific flow diagram of step S300;

[0047] Figure 4 Embodiment of the present invention Figure 3 Specific flow diagram of step S301;

[0048] Figure 5 Embodiment of the present invention Figure 3 Specific flow diagram of step S302;

[0049] Figure 6 It is a specific flow chart of a corpus similarity detection method provided by an embodiment of the present invention;

[0050] Figure 7 It is a structural schematic diagram of a corpus similarity detection device provided by an embodiment of the present invention;

[0051] Figure 8 It is a block diagram of the internal structure principle of an intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0053] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0054] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0055] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0056] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0057] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0059] With the development of science and technology, various large-scale models are gradually increasing. Large-scale pre-training models require a large amount of high-quality text corpus data as training samples. At present, most of the text corpus data used for training are crawled or collected from various data sources. There is a lot of redundant information in the collected data. Even for the same content, there may be differences between different data sources and different sections of the same data source due to editing problems, expression problems, or simple reprinting and modification. For example, the same news reported by different media may only differ in expression, headings or punctuation, but the actual content is mostly the same. Such similarities are more common than complete repetitions, and it is difficult to determine by direct equality comparison methods. If training samples need to obtain high-quality data, it is necessary to eliminate this redundant information to avoid the model accidentally over-learning content with high similarity. Therefore, corpus similarity detection is required to eliminate the corresponding redundant information. With the growing demand for data volume for large models, the scale of data that needs to be cleaned is also increasing. The traditional single-machine method is difficult to complete the processing within a controllable time limit. Therefore, in this embodiment, based on a distributed method, each computing node performs corpus similarity detection, thereby solving the problem of similar data detection of large-scale Chinese corpus, and then finding document pairs with relatively high similarity (where complete repetition can be considered to have a similarity of 100%), which facilitates subsequent analysis and cleaning of redundant data in the corpus based on the detection results to obtain high-quality training samples.

[0060] In the prior art, all documents are usually directly compared pairwise, which has a large amount of calculation (equivalent to the Cartesian product of the two), and the pairwise string comparison of long documents itself has a large amount of calculation and takes a long time. The problem of the prior art is that the solution of directly comparing all documents pairwise to realize corpus similarity detection has a large amount of calculation and takes a long time, which is not conducive to improving the efficiency of corpus similarity detection.

[0061] In one application scenario, similarity connection technology based on similarity threshold and local sensitive hashing technology can be used to detect corpus similarity. The similarity connection technology is to determine the similarity of any two documents by giving a similarity measurement distance definition (such as Jaccard distance) and a similarity threshold. If the measurement distance is within the threshold range, the two documents are determined to be similar; the local sensitive hashing technology is to process the original document with hash coding technology with document features, compress the document into a low-dimensional space for fuzzy similarity measurement. Most of these methods are single-machine processing methods, and it is difficult to obtain results within the expected time for large-scale data processing. Another method is Spark's MinHashLsh method. This method reduces the document dimension by using multiple hash functions to perform minimum hash coding on the original document set, and performs hash partitioning connections based on the coding value. The efficiency of similarity connection is improved at the cost of a small amount of precision. However, due to the randomness of the hash function and the presence of high-frequency words in the document, the load of some partitions is too high, and individual computing tasks often have serious tailing, which is particularly obvious when it comes to Chinese text data. The running efficiency is very low, and about 160MB of data needs to run for more than 10 hours, and the tailing task cannot be significantly improved by expanding computing resources or increasing parallelism.

[0062] In order to solve at least one of the above problems, in the scheme of the present invention, an input document set is obtained; each input document in the input document set is subjected to distributed preprocessing to obtain a set of coded documents to be detected, wherein the distributed preprocessing includes character removal, word segmentation and word format conversion, the set of coded documents to be detected includes multiple coded documents to be detected, and each word in the coded documents to be detected is an integer type code; based on the set of coded documents to be detected, corpus similarity detection is performed. Compared with the scheme of directly comparing documents by two strings in the prior art, the present invention first performs distributed preprocessing such as character removal, word segmentation, and word format conversion on the input documents to obtain a set of coded documents to be detected. Specifically, each coded document to be detected includes words after word segmentation, and each word has been converted into an integer type code. In this way, when comparing documents, only the words in the documents need to be compared respectively, and only integer type data comparison needs to be performed, which is conducive to reducing the amount of calculation, reducing the comparison time, and improving the efficiency of corpus similarity detection.

[0063] Exemplary Methods

[0064] like Figure 1 As shown, an embodiment of the present invention provides a corpus similarity detection method. Specifically, the method includes the following steps:

[0065] Step S100, obtaining an input document set.

[0066] The input document set is a set of documents that need to be tested for similarity, and the input document is the document that needs to be tested for similarity. In this embodiment, the document that needs to be tested for similarity is obtained from an external storage medium.

[0067] Step S200, respectively perform distributed preprocessing on each input document in the above-mentioned input document set to obtain a set of coded documents to be detected, wherein the above-mentioned distributed preprocessing includes character removal, word segmentation and word format conversion, the above-mentioned set of coded documents to be detected includes multiple coded documents to be detected, and each word in the above-mentioned coded documents to be detected is an integer type code.

[0068] The above-mentioned distributed preprocessing is a pre-set processing process, which can be set and adjusted according to actual needs. In the distributed preprocessing process, each input document can be processed in a distributed manner through different computing nodes to improve processing efficiency.

[0069] In this embodiment, the distributed preprocessing includes character removal, word segmentation and word format conversion. In actual application, other specific preprocessing processes may also be included, such as accurate deduplication, etc., which are not specifically limited here. Among them, word format conversion refers to converting words originally in the string format into an integer type encoding format, which is conducive to comparison and can improve the efficiency of comparison, thereby improving the efficiency of corpus similarity detection.

[0070] It should be noted that, in this embodiment, the above-mentioned distributed preprocessing process is performed based on the Spark distributed computing framework, and the above-mentioned corpus similarity detection method also runs on the Spark distributed computing framework, so it is necessary to pre-build and deploy the Spark computing platform.

[0071] Step S300: performing corpus similarity detection based on the above-mentioned set of coded documents to be detected.

[0072] Specifically, in this embodiment, the above-mentioned set of coded documents to be detected includes multiple coded documents to be detected, and it is necessary to perform corpus similarity detection on all the coded documents to be detected, and screen out documents with too high similarity, so as to remove redundant information. Wherein, each coded document to be detected includes multiple separated words, and each word is an integer type code, so it can be based on a distributed processing method, for two coded documents to be detected, each word is compared respectively, and all computing tasks will not be piled up on one computing node, which is conducive to improving processing efficiency. And in the process of comparing each word, only integer data needs to be compared, which can further improve processing efficiency compared to comparing character strings.

[0073] As can be seen from the above, in the corpus similarity detection method provided by the embodiment of the present invention, an input document set is obtained; each input document in the above input document set is distributedly preprocessed to obtain a set of coded documents to be detected, wherein the above distributed preprocessing includes character removal, word segmentation and word format conversion, the above coded document set to be detected includes multiple coded documents to be detected, and each word in the above coded document to be detected is an integer type code; corpus similarity detection is performed based on the above coded document set to be detected. Compared with the solution of directly comparing documents with each other in the prior art, the present invention first performs distributed preprocessing such as character removal, word segmentation, and word format conversion on the input document to obtain a set of coded documents to be detected. Specifically, each coded document to be detected includes words after word segmentation, and each word has been converted into an integer type code. In this way, when comparing documents, only the words in the documents need to be compared separately, and only integer type data comparison needs to be performed, which is conducive to reducing the amount of calculation, reducing the comparison time, and improving the efficiency of corpus similarity detection.

[0074] Specifically, in this embodiment, the above-mentioned input document is a Chinese document, and the data type of the above-mentioned input document is a string. In other application scenarios, the input document can also be a document corresponding to other languages, such as English documents, Japanese documents, etc., which are not specifically limited here. The above-mentioned input document set is a resilient distributed data set, and the above-mentioned distributed preprocessing and the above-mentioned corpus similarity detection are performed based on the Spark distributed framework. Specifically, the above-mentioned step S100 may include: reading the input data (i.e., input document set) InputRDD consisting of Chinese corpus data from external storage, InputRDD = RDD [original input document], wherein RDD represents Spark's resilient distributed data set, and what is actually stored in the data set is the original input document set, the original input document, i.e., the input document that needs to be detected by corpus (in this embodiment, the original input document represents the input document), and the document is of string type.

[0075] Specifically, in this embodiment, the above distributed preprocessing also includes global numbering, stop word removal and precise deduplication, such as Figure 2 As shown, the above step S200 includes:

[0076] Step S201, globally numbering each of the input documents in the input document set to obtain a numbered input document set.

[0077] Specifically, in this embodiment, the ZipWithIndex operation is performed on the input document set InputRDD to obtain the global number of the document, forming the numbered input document set IndexRDD, IndexRDD = RDD[(document number, original input document)], and persisted using the MemoryAndDisk strategy of Spark. In this way, each input document is globally numbered, and can be quickly mapped to the original input document after subsequent detection is completed, thereby removing redundant documents.

[0078] Step S202, respectively remove characters other than the target characters in the input documents of each number, and obtain documents of each target number, wherein the target characters are Chinese characters.

[0079] The target characters are characters that need to be detected for similarity, and the type of the target characters can be determined according to the type of corpus that needs to be detected for similarity. For a document, there may be multiple characters in the document, for example, a Chinese document may include Chinese characters, punctuation marks, and English characters. When Chinese corpus detection is required, the target characters are set to Chinese characters, so that other characters except Chinese characters can be deleted, and only Chinese characters are compared to improve processing efficiency.

[0080] Specifically, the Map operator of Spark is used to perform the conversion, and all characters other than Chinese characters in the input documents of each number are removed.

[0081] Step S203, segmenting each of the target numbered documents respectively to obtain a target word array corresponding to each of the target numbered documents.

[0082] Specifically, word segmentation is to divide the sentences in the document into individual words (i.e., words). After word segmentation, each word can be compared separately, which is conducive to improving processing efficiency. In this embodiment, the Map operator of Spark is used for Chinese word segmentation, and the Jieba word segmentation tool is used to divide each original input document in the numbered input document set into a word set. Specifically, each document is converted into a word array represented by the Array[String] type. In one embodiment, repeated words in the same document are recorded only once in the word array.

[0083] Step S204, broadcasting the stop word list to each computing node, and removing the stop words in each of the target word arrays based on the stop word list at each of the computing nodes to obtain a word array to be detected.

[0084] Among them, stop words are non-search words. In actual use, specific stop words can be set and adjusted according to actual needs, and are not specifically limited here. The stop word list is a table consisting of stop words that need to be eliminated. In this embodiment, the stop word list includes Chinese stop words. It should be noted that if the input document is an English document, when performing similarity detection on English corpus, what is included in the stop word list is English stop words, that is, the type of stop words needs to be determined according to the type of corpus for similarity detection.

[0085] In this embodiment, the Chinese stop word list is broadcast to each computing node, and the Map operator of Spark is used to remove the Chinese stop words in all target word arrays to obtain the corresponding word arrays to be detected.

[0086] Step S205, accurately remove duplicates from the set consisting of all the above-mentioned word arrays to be detected, and obtain a document set to be detected, wherein the above-mentioned document set to be detected includes multiple target detection word arrays.

[0087] Precise deduplication refers to removing identical documents. In this embodiment, precise deduplication is performed on each target number document after character removal (including removing characters other than the target character and stop words in the target character). Precise deduplication removes identical target number documents, which helps to reduce subsequent calculations and improve processing efficiency.

[0088] In this embodiment, the Distinct operator of Spark is used to accurately deduplicate the document set composed of word arrays, and the documents that are completely repeated after processing in the above steps are removed to obtain the document set to be detected SourceRDD, SourceRDD = RDD[(word array, document number)], and the word array included in the document set to be detected is the target detection word array.

[0089] Step S206, converting each word in each of the above target detection word arrays into an integer code, obtaining each target detection code array, and taking a set consisting of all target detection code arrays as the above code document set to be detected.

[0090] Specifically, each word in the target detection word array is of string type, and the amount of calculation when comparing is large, which affects the detection efficiency. Therefore, converting it into an integer type encoding can effectively reduce the amount of calculation when comparing.

[0091] In this embodiment, the above-mentioned code document set to be detected includes the target detection code array and the document number corresponding to each target detection code array, so as to be mapped to the corresponding original document. The Map conversion is performed on the document set SourceRDD to be detected, and the words are converted from String type to Int type encoding according to the MurmurHash algorithm, and the document number is recorded to form the code document set HashRDD to be detected, HashRDD = RDD [(word set, document number)], where the word set is each target detection code array, the word set is Array [Int] type, and the document number is Long type.

[0092] In this way, based on the above distributed preprocessing process, the number of potentially similar documents can be significantly reduced, the computational cost of the similarity verification step can be reduced, and the word string can be mapped to a 32-bit integer encoding ID. After the conversion, the subsequent similarity calculation steps will be changed from a more complex string comparison to an integer comparison, which reduces the computational cost. In addition, all Chinese words are compressed to 4 bytes, which improves the reading, writing and communication efficiency to a certain extent, thereby improving the efficiency of similarity detection.

[0093] Specifically, in this embodiment, Figure 3 As shown, the above step S300 includes:

[0094] Step S301: construct a target word frequency table based on the above-mentioned set of coded documents to be detected.

[0095] Step S302: performing similarity detection on the above-mentioned set of coded documents to be detected based on the above-mentioned target word frequency table.

[0096] The target word frequency table is a table constructed based on the set of coded documents to be detected and is used to reflect the frequency of occurrence of each word in the document.

[0097] Specifically, in this embodiment, Figure 4 As shown, the above step S301 specifically includes:

[0098] Step S3011, expand the word set of the above-mentioned coded document set to be detected to obtain an initial word frequency table, which includes A tuples, each tuple includes a word and the word frequency corresponding to the word, the above A is the number of words in the above-mentioned coded document set to be detected, and the frequency of each word in the above-mentioned initial word frequency table is 1.

[0099] Specifically, the initial word frequency table includes every word in the set of coded documents to be detected (the same word can appear repeatedly), and the initial word frequency of each word is 1, which is convenient for subsequent merge statistics. In this embodiment, the FlatMap operator is executed on the set of coded documents to be detected HashRDD to expand the word set to obtain the initial word frequency table FlatHashRDD, FlatHashRDD = RDD [(word code, word frequency)], and the word frequency is initialized to 1. Step S3012, merge the tuples with the same word code in the above initial word frequency table, and sum the corresponding word frequencies to obtain a complete word frequency table.

[0100] Among them, the tuple represents a pair of (word code, word frequency) in the initial word frequency table FlatHashRDD. When the word codes are the same, it means that the corresponding words are the same and the word appears repeatedly. At this time, the same word codes can be merged and one word code is left. The corresponding word frequencies are summed up to quickly count the frequency of each word and obtain a complete word frequency table.

[0101] Specifically, in this embodiment, a ReduceByKey operation is performed on the initial word frequency table FlatHashRDD, and for tuples with the same word code, the word frequencies are merged and summed to obtain a complete word frequency table WordRDD, WordRDD = RDD[(word code, word frequency)], where the word code and word frequency are both of Int type.

[0102] Step S3013, obtaining a target word frequency table after deleting low-frequency tuples in the above complete word frequency table, wherein the word frequency of the above low-frequency tuples is less than or equal to a preset word frequency threshold.

[0103] In one application scenario, the complete word frequency table can be directly used as the target word frequency table for subsequent corpus similarity detection. In another application scenario, a delayed calculation strategy based on the word frequency table can also be adopted. Specifically, in the word frequency statistics process, a delayed calculation strategy is adopted to not record the word frequency of low-frequency words. In the subsequent word frequency sorting and similarity calculation, the words without recorded word frequency are uniformly calculated as 1. The above-mentioned delayed calculation strategy can greatly reduce the communication volume and memory usage when aggregating word frequencies in partitions, thereby improving processing efficiency. In actual use, you can only not record words with a word frequency of 1, or eliminate low-frequency words with a word frequency of >1, and you can use a preset word frequency threshold to limit it.

[0104] Among them, the above-mentioned preset word frequency threshold is a pre-set threshold for distinguishing low-frequency words. In this embodiment, it is set to 1. During actual use, it can also be set and adjusted according to actual needs, and no specific limitation is made here.

[0105] In this embodiment, in order to reduce the amount of communication and memory usage, the target word frequency table is obtained after deleting the low-frequency tuples in the complete word frequency table. Specifically, the Filter operation is performed on the complete word frequency table WordRDD to delete the low-frequency word tuples therein (in this embodiment, only the tuples with word frequency = 1 are deleted, which can be adjusted according to the specific situation to obtain a higher compression rate). It should be noted that the process of obtaining the complete word frequency table and the target word frequency table can be processed in partitions, and then CollectAsMap is used to aggregate the filtered word tables of each partition to form the target word frequency table WordMap, WordMap = Map[word code->word frequency], where word code and word frequency are both of Int type. After obtaining the target word frequency table, it can be broadcast to each computing node for the next step of calculation.

[0106] Specifically, in this embodiment, Figure 5 As shown, the above step S302 specifically includes:

[0107] Step S3021, sorting the words in each of the above target detection coding arrays in the above code document set to be detected according to the word frequency in the above target word frequency table, and obtaining each sorted coding array.

[0108] Specifically, in this embodiment, a Map conversion is performed on the HashRDD set of coded documents to be detected, and the words in the word array are sorted from small to large according to the word frequency in the target word frequency table WordMap to form multiple sorted coded arrays, and the set composed of all sorted coded arrays is a sorted coded array set SortedHashRDD, SortedHashRDD = RDD[(word array, document number)]. Among them, each word array in SortedHashRDD is a sorted sorted coded array.

[0109] Step S3022, respectively obtain the prefix word arrays corresponding to the above-mentioned sorted code arrays.

[0110] Specifically, in this embodiment, Map transformation is performed on SortedHashRDD, and the prefix calculation formula is used to calculate the prefix length of each document x to obtain a prefix word array. The type of the prefix word array is Array[Int], which is a subarray intercepted from the word array according to the calculated prefix length. The above prefix calculation formula is Where x represents the document (i.e., the sorted encoding array), j represents the prefix length, i is an integer between 1 and j, and weight(x[i]) represents the IDF weight of the i-th word in document x, where x[i] represents the i-th word in document x, σ x Represents the sum of the IDF weights of each word in document x. The IDF weight of a word can be calculated based on the word frequency and the total number of records: Among them, idf i represents the IDF weight of a word (i-th word), D represents the number of documents in the corpus, j represents a document number in the corpus, and t i Represents any word, d j Represents the word t i A document of x is the idf of all words in a document (i.e. a sorted encoded array) i The sum of .

[0111] Step S3023, partitioning and grouping the target detection encoding array based on the prefix word array.

[0112] In this embodiment, the target detection encoding array is partitioned and grouped, so that distributed processing is performed through Spark, and each partition and each group in each partition can be processed separately, which is conducive to improving processing efficiency. Among them, a partition can include multiple groups or only one group, which is not specifically limited here.

[0113] Specifically, traverse each word in the prefix word array, and perform FlatMap operation in the manner of (word code, word array) to expand it to form an expanded data set PrefixRDD, PrefixRDD = RDD[(word code, word array, document number)], wherein the word array in the expanded data set is the word array corresponding to the input document (and the document number) after distributed preprocessing and encoding of the encoded document to be detected, that is, the target detection encoding array, not the prefix word array.

[0114] The above-mentioned target detection coding array is partitioned and grouped based on the above-mentioned prefix word array, including: dividing the target detection coding array with at least one common word in the prefix into the same group of the same partition of Spark. Specifically, the PrefixRDD is subjected to a GroupByKey operation, and the partitioning and grouping are performed according to the word code, which is equivalent to aggregating the documents (i.e., the target detection coding array) with at least one common word in the prefix into the same partition group of Spark, forming a partitioned grouped data set GroupPrefixRDD, GroupPrefixRDD = RDD[word code, Iterable[(word set, document number)]], wherein each Iterable represents a group, each group contains the same word code, and the word set in the Iterable is the target detection coding array. The processing efficiency can be improved by partitioning and grouping, and it is convenient to perform distributed computing on the similarity verification step.

[0115] Step S3024, performing similarity verification based on the partitioned and grouped target detection coding array.

[0116] Specifically, the above-mentioned similarity verification based on the target detection coding array after partitioning and grouping includes: based on a preset similarity calculation formula, performing similarity verification on the target detection coding array in each group respectively, and obtaining the document number corresponding to the similarity detection coding array, wherein the above-mentioned similarity detection coding array is a target detection coding array that meets the preset similarity condition; based on the above-mentioned document number, obtaining the corresponding input document that meets the preset similarity condition.

[0117] Furthermore, after similarity verification, result assembly can also be performed. Specifically, based on each computing node, similarity verification is performed on the GroupPrefixRDD in the above partitioned grouped data set. A Map operation is performed on the GroupPrefixRDD, and similarity verification is performed on the records in each group according to the similarity calculation formula, and each partition performs distributed parallel calculation independently. The above similarity calculation formula is a preset formula and can be adjusted according to actual needs. In one embodiment, the similarity calculation formula is specifically the Jaccard similarity formula: Among them, J(a,b) represents the similarity between documents a and b, ω represents a word that meets the conditions, a represents document a, b represents document b, and weight(ω) represents the weight of word ω.

[0118] In an application scenario, before the verification calculation, the length pruning and position pruning schemes in the traditional single-machine method can be introduced to further reduce the verification cost, and finally obtain the duplicate record pair DuplicateRDD that meets the similarity condition, DuplicateRDD = RDD [(document number, document number)], where the two document numbers in DuplicateRDD correspond to the target detection encoding array that meets the similarity condition, and because they are global numbers, the original input documents can be obtained according to the corresponding document numbers, thereby eliminating redundant documents. It should be noted that the similarity condition can be preset. For example, if the calculated similarity is greater than the preset similarity threshold, the two are considered to be duplicate documents. The specific similarity threshold can be set and adjusted according to actual needs, and is not specifically limited here.

[0119] Furthermore, the above DuplicateRDD and the number after global numbering are input into the document collection IndexRDD for Join operation, and the keyword of Join is the document number, so as to obtain the real duplicate document detection result, and save the result to external storage to eliminate redundant documents.

[0120] In this embodiment, the above corpus similarity detection method is further described in detail based on a specific application scenario. Figure 6 1 is a specific flow chart of a corpus similarity detection method provided by an embodiment of the present invention. Specifically, the above method is based on the Spark distributed memory computing framework and combines the characteristics of Chinese data to perform similarity detection on large-scale Chinese corpus data. Figure 6 As shown in the figure, in the above method, the original Chinese corpus data is preprocessed in a targeted manner using the Spark framework. Immediately after reading the data, the document collection is distributedly numbered, and then non-Chinese symbols in the corpus are removed to avoid the influence of high-frequency words composed of symbols. The document is segmented by using the stuttering method and then the stop words are removed to further reduce the high-frequency words. Since a lot of content has been removed from the document, and the document itself may also be completely repeated, on this basis, the document collection is preprocessed by using the precise deduplication method. This set of preprocessing processes can significantly reduce the number of potential similar documents and reduce the computational cost of the similarity verification step. Finally, for each word in each document, the MurmurHash hashing algorithm is embedded in the distributed Map operator of Spark to map the word string to a 32-bit integer encoding ID. After the conversion, the subsequent similarity calculation step will be changed from a string comparison with a high complexity to an integer comparison, which reduces the computational cost, and all Chinese words are compressed to 4 bytes, which improves the reading, writing and communication efficiency to a certain extent. Furthermore, a delayed calculation strategy is adopted for the processing of the word frequency table. Spark is used to perform distributed word frequency statistics on the corpus data and convert it into a word frequency table. In the word frequency statistics process, a delayed calculation strategy is adopted. For low-frequency words, their word frequencies are not recorded. In the subsequent word frequency sorting and similarity calculation, the words without word frequency records are uniformly calculated as 1. Statistical experiments have found that a large number of Chinese words have low word frequencies. This strategy can greatly reduce communication volume and memory usage when the word frequencies are summarized in partitions. The number of Chinese word segmentation words is very large compared to English. For example, the number of words in the 220 million CommonCrawl corpus data after the word segmentation by Jieba reaches 130 million, which consumes huge memory and communication volume. In this embodiment, only 47 million words are compressed according to this method. In addition, in this embodiment, only word records with word frequency = 1 are not recorded. In actual use, low-frequency word records with word frequency > 1 can also be eliminated. The word frequency table will be further reduced, but the accuracy will be slightly reduced, but it can cope with larger data sets. It can be selected according to the specific situation.

[0121] Further, a prefix-based partitioning and parallel computing strategy is also adopted in this embodiment. It should be noted that, given a string arranged in any order, if another string is also arranged in the same order, then at least one word in the other string appears in the prefix of the given string. According to this prefix rule, the words of each document are arranged in order from small to large according to the frequency of the word, and the prefix part of each document is calculated, and each document is subjected to the partition distribution process of Spark in the form of (word encoding, word set) according to each word in the prefix. Documents with common words in the prefix will be gathered in a partition group. This method filters a large number of record pairs that do not meet the prefix rule, greatly reducing the amount of calculation for subsequent verification. At the same time, since the documents are arranged in increasing order of frequency according to the Chinese vocabulary, the words that appear in the prefix are generally relatively low-frequency words, so there will not be too many documents falling in a partition, and compared with the MinHash partitioning scheme of Spark, the tailing phenomenon will also be significantly reduced. Through this process, all documents are distributed to different partition groups, and each group contains potentially similar documents. Finally, a similarity comparison and verification is performed between each group. Before comparison, the length information and word position information of the record are used for basic judgment and filtering. For document pairs that cannot be filtered out, the document similarity is calculated pairwise to obtain document number pairs that meet the similarity. Finally, the original records are connected according to the number and output as duplicate document detection results.

[0122] Thus, in this embodiment, a fully distributed corpus similarity detection method based on Spark is provided, and all steps thereof have parallel and scalable characteristics, and are suitable for similarity detection of large-scale Chinese corpus data. Among them, there are preprocessing steps for the characteristics of large-scale Chinese corpus data, such as distributed document numbering, elimination of non-Chinese symbols, elimination of word segmentation and stop words, pre-deduplication (i.e., accurate deduplication) and word encoding, which significantly reduce the tailing phenomenon caused by high-frequency words, data reading and writing and communication costs, and subsequent similarity verification calculations.

[0123] In the process of constructing the word frequency table, it is constructed according to the characteristics of Chinese data. The Chinese data word table is relatively large. The delayed calculation strategy is used for the word frequency calculation of low-frequency words. This can significantly compress the size of the word frequency table and solve the problem that the Chinese word frequency table is too large to be accommodated in the memory. At the same time, it also reduces the communication volume, allowing the algorithm to adapt to the data scale of billions.

[0124] Furthermore, prefix-based partitioning and grouping and Spark-based parallel computing methods are also provided. Documents are partitioned and grouped based on common words, and the number of distributions is reduced through prefix pruning, word array pre-sorting, and IDF weight semantics, thereby reducing the communication cost and computing cost of the entire computing process. Compared with Spark's MinhashLsh partitioning scheme, it can make better use of the semantic characteristics of similarity for calculation, further reducing the tailing phenomenon, and the algorithm has a higher degree of parallelism.

[0125] Specifically, compared with the traditional Simhash scheme and Spark's MinhashLsh scheme, distributed similarity detection is realized without hash compression of documents. Finally, the complete Jaccard similarity measurement formula is used for verification, and the similarity detection accuracy is higher.

[0126] Exemplary Devices

[0127] like Figure 7 As shown in , corresponding to the above-mentioned corpus similarity detection method, an embodiment of the present invention further provides a corpus similarity detection device, and the above-mentioned corpus similarity detection device includes:

[0128] The input document set acquisition module 410 is used to acquire an input document set.

[0129] The input document set is a set of documents that need to be tested for similarity, and the input document is the document that needs to be tested for similarity. In this embodiment, the document that needs to be tested for similarity is obtained from an external storage medium.

[0130] The distributed preprocessing module 420 is used to perform distributed preprocessing on each input document in the above-mentioned input document set respectively to obtain a set of coded documents to be detected, wherein the above-mentioned distributed preprocessing includes character removal, word segmentation and word format conversion, the above-mentioned set of coded documents to be detected includes multiple coded documents to be detected, and each word in the above-mentioned coded documents to be detected is an integer type code.

[0131] The above-mentioned distributed preprocessing is a pre-set processing process, which can be set and adjusted according to actual needs. In the distributed preprocessing process, each input document can be processed in a distributed manner through different computing nodes to improve processing efficiency.

[0132] In this embodiment, the distributed preprocessing includes character removal, word segmentation and word format conversion. In actual application, other specific preprocessing processes may also be included, such as accurate deduplication, etc., which are not specifically limited here. Among them, word format conversion refers to converting words originally in the string format into an integer type encoding format, which is conducive to comparison and can improve the efficiency of comparison, thereby improving the efficiency of corpus similarity detection.

[0133] It should be noted that, in the present embodiment, the above-mentioned distributed preprocessing process is performed based on the Spark distributed computing framework, and the Spark computing platform can be pre-built and deployed in the above-mentioned corpus similarity detection device.

[0134] The corpus similarity detection module 430 is used to perform corpus similarity detection based on the above-mentioned set of coded documents to be detected.

[0135] Specifically, in this embodiment, the above-mentioned set of coded documents to be detected includes multiple coded documents to be detected, and it is necessary to perform corpus similarity detection on all the coded documents to be detected, and screen out documents with too high similarity, so as to remove redundant information. Wherein, each coded document to be detected includes multiple separated words, and each word is an integer type code, so it can be based on a distributed processing method, for two coded documents to be detected, each word is compared respectively, and all computing tasks will not be piled up on one computing node, which is conducive to improving processing efficiency. And in the process of comparing each word, only integer data needs to be compared, which can further improve processing efficiency compared to comparing character strings.

[0136] As can be seen from the above, in the corpus similarity detection device provided by the embodiment of the present invention, the input document set is obtained by the input document set acquisition module 410; the distributed preprocessing module 420 performs distributed preprocessing on each input document in the above input document set to obtain the coded document set to be detected, wherein the above distributed preprocessing includes character removal, word segmentation and word format conversion, the above coded document set to be detected includes multiple coded documents to be detected, and each word in the above coded document to be detected is an integer type code; the corpus similarity detection module 430 performs corpus similarity detection based on the above coded document set to be detected. Compared with the solution of directly comparing documents with each other in the prior art, the present invention first performs distributed preprocessing such as character removal, word segmentation, and word format conversion on the input document to obtain the coded document set to be detected. Specifically, each coded document to be detected includes words after word segmentation, and each word has been converted into an integer type code. In this way, when comparing documents, only the words in the documents need to be compared separately, and only integer type data comparison needs to be performed, which is conducive to reducing the amount of calculation, reducing the comparison time, and improving the efficiency of corpus similarity detection.

[0137] Specifically, in this embodiment, the specific functions of the above-mentioned corpus similarity detection device and its various modules can refer to the corresponding description in the above-mentioned corpus similarity detection method, and will not be repeated here.

[0138] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 8 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a corpus similarity detection program. The internal memory provides an environment for the operation of the operating system and the corpus similarity detection program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the corpus similarity detection program is executed by the processor, the steps of any one of the above-mentioned corpus similarity detection methods are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0139] Those skilled in the art will understand that Figure 8 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] In one embodiment, a smart terminal is provided. The smart terminal includes a memory, a processor, and a corpus similarity detection program stored in the memory and executable on the processor. When the corpus similarity detection program is executed by the processor, the following operation instructions are performed:

[0141] Get the input document collection;

[0142] Performing distributed preprocessing on each input document in the input document set to obtain a set of coded documents to be detected, wherein the distributed preprocessing includes character removal, word segmentation and word format conversion, the set of coded documents to be detected includes multiple coded documents to be detected, and each word in the coded documents to be detected is an integer type code;

[0143] Corpus similarity detection is performed based on the above-mentioned set of coded documents to be detected.

[0144] An embodiment of the present invention further provides a computer-readable storage medium, on which a corpus similarity detection program is stored. When the corpus similarity detection program is executed by a processor, the steps of any one of the corpus similarity detection methods provided in the embodiments of the present invention are implemented.

[0145] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0146] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0147] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0148] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0149] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0150] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0151] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A corpus similarity detection method, characterized in that: The method comprises: Get the input document collection; Performing distributed preprocessing on each input document in the input document set respectively to obtain a set of coded documents to be detected, wherein the distributed preprocessing includes character removal, word segmentation and word format conversion, the set of coded documents to be detected includes multiple coded documents to be detected, and each word in the coded documents to be detected is an integer type code; Performing corpus similarity detection based on the set of coded documents to be detected; Wherein, the input document set is a resilient distributed data set, and the distributed preprocessing and the corpus similarity detection are performed based on the Spark distributed framework; The distributed preprocessing also includes global numbering, stop word removal and precise deduplication. The distributed preprocessing is performed on each input document in the input document set to obtain a set of coded documents to be detected, including: Globally numbering each input document in the input document set to obtain a numbered input document set; Respectively remove characters other than the target character from each numbered input document to obtain each target numbered document, wherein the target character is a Chinese character; Perform word segmentation on each of the target numbered documents respectively to obtain a target word array corresponding to each of the target numbered documents; Broadcasting a stop word list to each computing node, and removing stop words in each target word array based on the stop word list at each computing node to obtain a word array to be detected; Accurately remove duplicates from the set consisting of all the word arrays to be detected to obtain a document set to be detected, wherein the document set to be detected includes multiple target detection word arrays; Each word in each of the target detection word arrays is converted into an integer code, each target detection code array is obtained, and a set consisting of all target detection code arrays is used as the set of code documents to be detected.

2. The corpus similarity detection method according to claim 1, characterized in that: The input document is a Chinese document.

3. The corpus similarity detection method according to claim 1, characterized in that: The performing corpus similarity detection based on the set of coded documents to be detected includes: Constructing a target word frequency table based on the set of coded documents to be detected; The set of coded documents to be detected is subjected to similarity detection based on the target word frequency table.

4. The corpus similarity detection method according to claim 3, characterized in that: The step of constructing a target word frequency table based on the set of coded documents to be detected includes: Expand the word set of the coded document set to be detected to obtain an initial word frequency table, wherein the initial word frequency table includes A tuples, each tuple includes a word and a word frequency corresponding to the word, wherein A is the number of words in the coded document set to be detected, and each word frequency in the initial word frequency table is 1; Merge tuples with the same word code in the initial word frequency table, and sum the corresponding word frequencies to obtain a complete word frequency table; The target word frequency table is obtained after deleting the low-frequency tuples in the complete word frequency table, wherein the word frequency of the low-frequency tuples is less than or equal to a preset word frequency threshold.

5. The corpus similarity detection method according to claim 4, characterized in that: The preset word frequency threshold is 1.

6. The corpus similarity detection method according to claim 3, characterized in that: The performing similarity detection on the set of coded documents to be detected based on the target word frequency table includes: Sort the words in each of the target detection coding arrays in the set of coded documents to be detected according to the word frequencies in the target word frequency table to obtain each sorted coding array; Respectively obtain the prefix word arrays corresponding to the sorted encoding arrays; Partitioning and grouping the target detection encoding array based on the prefix word array; Similarity verification is performed based on the partitioned and grouped target detection encoding array.

7. The corpus similarity detection method according to claim 6, characterized in that: The partitioning and grouping of the target detection encoding array based on the prefix word array includes: dividing the target detection encoding array having at least one common word in the prefix into the same group of the same partition of Spark.

8. The corpus similarity detection method according to claim 7, characterized in that: The similarity verification based on the partitioned and grouped target detection encoding array includes: Based on a preset similarity calculation formula, similarity verification is performed on the target detection code arrays in each group, and a document number corresponding to the similar detection code array is obtained, wherein the similar detection code array is a target detection code array that meets a preset similarity condition; Based on the document number, a corresponding input document that meets a preset similarity condition is obtained.

9. A corpus similarity detection device, characterized in that: The device comprises: An input document set acquisition module is used to acquire an input document set; A distributed preprocessing module is used to perform distributed preprocessing on each input document in the input document set to obtain a set of coded documents to be detected, wherein the distributed preprocessing includes character removal, word segmentation and word format conversion, the set of coded documents to be detected includes multiple coded documents to be detected, and each word in the coded documents to be detected is an integer type code; A corpus similarity detection module, used for performing corpus similarity detection based on the set of coded documents to be detected; Wherein, the input document set is a resilient distributed data set, and the distributed preprocessing and the corpus similarity detection are performed based on the Spark distributed framework; The distributed preprocessing also includes global numbering, stop word removal and precise deduplication. The distributed preprocessing is performed on each input document in the input document set to obtain a set of coded documents to be detected, including: Globally numbering each input document in the input document set to obtain a numbered input document set; Respectively remove characters other than the target character from each numbered input document to obtain each target numbered document, wherein the target character is a Chinese character; Perform word segmentation on each of the target numbered documents respectively to obtain a target word array corresponding to each of the target numbered documents; Broadcasting a stop word list to each computing node, and removing stop words in each target word array based on the stop word list at each computing node to obtain a word array to be detected; Accurately remove duplicates from the set consisting of all the word arrays to be detected to obtain a document set to be detected, wherein the document set to be detected includes multiple target detection word arrays; Each word in each of the target detection word arrays is converted into an integer code, each target detection code array is obtained, and a set consisting of all target detection code arrays is used as the set of code documents to be detected.

10. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a corpus similarity detection program stored in the memory and executable on the processor. When the corpus similarity detection program is executed by the processor, the steps of the corpus similarity detection method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a corpus similarity detection program, which, when executed by a processor, implements the steps of the corpus similarity detection method according to any one of claims 1 to 8.

Images