Text comparison method based on sliding window and dynamically expanded KMP algorithm
By combining a sliding window with the dynamically extended KMP algorithm, the contradiction between matching accuracy and efficiency in traditional text similarity detection is resolved, achieving efficient and accurate text similarity detection, which is suitable for complex document structures and large-scale document comparison.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU JIUTIAN ZHIFEI TECH CO LTD
- Filing Date
- 2025-09-05
- Publication Date
- 2026-05-26
AI Technical Summary
Existing text similarity detection algorithms suffer from a trade-off between matching accuracy and performance when dealing with large-scale complex documents. In particular, it is difficult to balance the identification of repetitive features at multiple scales with computational efficiency. Traditional methods result in the loss of location information, high false negative rates, and low efficiency.
A text matching method based on sliding window and dynamically extended KMP algorithm is adopted. Substrings are extracted by sliding window with step size of 1, an LPS array is constructed, and matching is performed by the improved dynamically extended KMP algorithm. Multi-threaded parallel processing and secondary verification are used to ensure the accuracy of the results.
It achieves high-precision duplicate content recognition in complex text environments, solves the problems of lost location information and high false negative rate in traditional methods, improves matching efficiency and result accuracy, adapts to multi-format text structures and supports large-scale document comparison needs.
Smart Images
Figure CN121145834B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of document processing and text analysis technology, specifically a text comparison method based on a sliding window and dynamically extended KMP algorithm. Background Technology
[0002] With the explosive growth in the scale of digital documents, the demand for efficient and accurate text similarity detection across different domains is becoming increasingly urgent. However, current mainstream technical solutions all have significant bottlenecks: traditional brute-force matching algorithms have extremely high time complexity due to exhaustive character-by-character comparison, making them unsuitable for large-scale data scenarios; the Rabin-Karp algorithm, although introducing a hash acceleration mechanism, is limited by the characteristic that the probability of hash collisions increases with the length of the text, and also consumes a lot of memory; the standard KMP algorithm can only locate the starting point of a single match, lacks dynamic expansion capabilities, and cannot effectively capture long-span repetitive paragraphs. Techniques optimized for specific scenarios also have inherent defects—sliding window-based methods, due to their fixed window width design, may miss long repetitive segments due to an excessively small window, and are difficult to adapt to variable-length repetitive content due to the lack of a dynamic adjustment mechanism; the improved KMP algorithm, although improving the efficiency of a single match through the LPS array, is still limited to static matching of a single pattern string and cannot automatically expand to discover continuous repetitive regions; hash-based approximate matching suffers from the dual limitations of increased false positive rates caused by hash collisions and the inability to achieve character-level precise matching, making it difficult to meet the requirements for high-precision duplicate detection. The common problem with the aforementioned technologies is their failure to balance multi-scale repetitive feature recognition with computational efficiency. This is particularly evident when dealing with complex document structures, where a trade-off between matching accuracy and performance is prevalent. Therefore, this invention proposes a text matching method based on a sliding window and a dynamically extended KMP algorithm. Summary of the Invention
[0003] (a) Technical problems to be solved
[0004] To address the shortcomings of existing technologies, this invention provides a text comparison method based on a sliding window and a dynamically extended KMP algorithm. This method is characterized by high efficiency, accuracy, and practicality, and solves the problems of loss of original location information, high false negative rate, and low comparison efficiency caused by preprocessing in traditional document similarity detection.
[0005] (II) Technical Solution
[0006] To achieve the above objectives, the present invention provides the following technical solution: a text comparison method based on a sliding window and dynamically extended KMP algorithm, comprising the following steps:
[0007] Step 1: Set algorithm parameters and prepare input documents;
[0008] Step 2: Extract substrings from the document to be detected using a sliding window with a step size of 1;
[0009] Step 3: Construct an LPS array containing the original position offsets of valid characters;
[0010] Step 4: Use the improved dynamically expanding KMP algorithm to perform duplicate content matching;
[0011] Step 5: Store the matching results in a hierarchical data structure in real time;
[0012] Step 6: Start multi-threading to perform deduplication, merging, and validation operations on the matching results;
[0013] Step 7: Ensure the accuracy of the merged results through secondary verification;
[0014] Step 8: Output structured text comparison results.
[0015] Preferably, the steps for setting algorithm parameters and preparing input documents in step one are as follows:
[0016] S1.1 First, determine the minimum number of matching characters threshold N. This threshold can be adjusted according to the specific application scenario.
[0017] S1.2. Then, simultaneously acquire two input documents to be compared, retaining all original character structures in the documents, including spaces, full-width / half-width punctuation, tabs and newlines, and other special characters, without performing pre-cleaning processing.
[0018] Preferably, in step two, a sliding window with a step size of 1 is used to extract the substring of the document to be detected. The specific operation is as follows:
[0019] S2.1. The initial size of the sliding window is set as the minimum number of matching characters threshold N in step one;
[0020] S2.2. Starting from the beginning of the first document, scan the document character by character using a step size of 1;
[0021] S2.3 After each move, extract a substring of length N as the pattern string until the end of the window exceeds the document length. During the extraction process, strictly preserve the original character structure of the substring.
[0022] Preferably, the process of constructing the LPS array in step three is as follows:
[0023] S3.1. Construct the LPS array required by the KMP algorithm for the pattern string extracted in step two;
[0024] S3.2 During the construction process, the position offset of each valid character in the original pattern string is recorded synchronously.
[0025] Preferably, step four uses an improved dynamically extended KMP algorithm for duplicate content matching, which specifically includes three parts:
[0026] (1) Real-time character filtering in the matching stage: In the character comparison step of KMP, when the character to be matched is a special symbol, the algorithm does not interrupt the matching process, but directly moves the current character pointer to the next non-special character and continues to perform the comparison;
[0027] (2) Initial matching judgment: When the pattern string and the target document fragment are compared through real-time character filtering, if the matching length is ≥ the minimum number of matching characters threshold N set in step one, the initial matching is judged to be successful; if the matching fails, the sliding window is moved in the manner of step two to continue to extract new pattern strings for matching.
[0028] (3) Dynamic expansion matching: After the initial matching is successful, the algorithm automatically enters the expansion stage. Starting from the current matching endpoint, the pointers of the pattern string and the target document are moved backward to continue the real-time character filtering mechanism. For each valid character expanded, the original position information of the matching segment is updated. The expansion terminates when three consecutive valid characters do not match or any document reaches the end, and finally the longest repeated content segment is determined.
[0029] Preferably, in step five, the matching results are stored in real time to a hierarchical data structure. The hierarchical data structure is used to store the matching results obtained in step four. This data structure includes a Result class and a MatchResult class.
[0030] Preferably, the Result class stores: the core information of a single matching result, including the starting position start1_ in the original document 1 and the starting position start2_ in the document 2, the effective character length and original length of the matching segment, and the original string of the matching content;
[0031] The MatchResult class stores information by inheriting from the Result class and extending its collection storage capabilities. It records the position information of multiple matching segments through the start1_ and start2_ lists and provides the AddRe method for temporarily merging adjacent matching results.
[0032] Preferably, step six, which involves starting multiple threads to perform deduplication, merging, and validation operations on the matching results, specifically includes:
[0033] S4.1 Multi-threaded parallel processing settings: Parallelization is achieved using a producer-consumer model. After the main thread generates the matching results in step five, it stores the Result object in a thread-safe result queue. An independent merging thread extracts the results from the queue and performs subsequent deduplication and merging operations without blocking the matching process of the main thread. When the document length exceeds 10,000 characters, multiple merging threads are automatically started and scheduled through dynamic priority.
[0034] S4.2 Result Deduplication: Handle inclusion relationships. When the original location range of result A completely covers result B, keep A and discard B.
[0035] S4.3 Result Merging: Handling overlapping relationships. When the overlap length of two results accounts for ≥60% of the original length of the shorter result, and the effective character matching rate is ≥95%, they are merged into one record. That is, the starting position is taken as the smaller value, the ending position is taken as the larger value, and the original length is updated to the actual length after merging.
[0036] Preferably, the secondary verification operation in step seven is as follows:
[0037] S5.1. For the merged result in step six, the complete merged segment is re-matched using the improved KMP algorithm to verify whether the merged content is completely consistent after ignoring symbols.
[0038] S5.2 Simultaneously, combined with the sliding window verification mechanism, a sliding window with a step size of 1 is used to cover the merged matching results and calculate the character density of the matching fragment. When the character density threshold is ≥80%, the merged result is determined to be valid; otherwise, the KMP algorithm is triggered to rescan.
[0039] Preferably, in step eight, text comparison involves integrating the merged results verified in step seven and outputting structured information, including: the original start / end positions of the merged duplicate segments in the two documents, the effective character length of the duplicate segments and the original text content, and the proportion of the duplicate segments in the documents.
[0040] Compared with existing technologies, this invention provides a text comparison method based on a sliding window and a dynamically extended KMP algorithm, which has the following advantages:
[0041] 1. This invention achieves high-precision recognition of repetitive content in complex text environments through a collaborative mechanism of sliding window and dynamically expanded KMP algorithm. By employing a fine scanning window with a step size of 1, it ensures that no potential repetitive segments are missed. Combined with the improved KMP algorithm, it skips invalid characters such as spaces and punctuation in real time during the matching process. This not only preserves the symbol position information of the original document but also avoids the impact of symbol differences on matching accuracy. Through dynamically expanded matching logic and secondary verification mechanism, it can accurately locate the longest repetitive paragraph and eliminate misjudgments caused by accidental matching, greatly improving the recognition accuracy of this invention in complex text scenarios.
[0042] 2. This invention overcomes the limitations of traditional algorithms in adapting to multi-format text by using real-time character filtering and position mapping technology. Based on a dynamic character type recognition mechanism within the ASCII code range, it can be compatible with complex text structures containing mixed Chinese and English characters and full-width / half-width symbols without preprocessing. It also records the original position offset of skipped symbols simultaneously, enabling the matching results to be accurately mapped to the original document index. This solves the problem of matching breaks caused by symbol interference in traditional methods and enhances the adaptability of this invention to legal documents and bidding documents containing special format texts.
[0043] 3. This invention constructs an efficient and stable document comparison pipeline through modular design and multi-threaded parallel processing, decoupling the matching process from the result processing. It adopts a producer-consumer model to achieve parallel execution of matching and merging. By combining sliding windows with KMP jump logic, redundant calculations are reduced. The independently running merging thread ensures concise and reliable results through intelligent deduplication, overlapping merging, and secondary verification. This architecture can support dynamic adjustment of threshold parameters and is suitable for large-scale document comparison needs in the fields of education, publishing, and intellectual property.
[0044] 4. This invention establishes a highly reliable content similarity detection standard through a zero-false-error rate guarantee system. It forms a three-level quality control mechanism, from character-level filtering to dynamic expansion matching, and then to KMP secondary verification after result merging. The sliding window verification mechanism forces the matching fragments to meet the character density standard to completely eliminate false matches caused by symbol filtering. The merging rules ensure the accuracy and consistency of the output results through dual judgment of position range analysis and content similarity, ultimately meeting the high rigor requirements of scenarios such as financial contracts and scientific research papers. Attached Figure Description
[0045] Figure 1 This is a flowchart illustrating the overall workflow of the present invention;
[0046] Figure 2 This invention stores a class diagram of the results.
[0047] Figure 3 This is a flowchart of the result merging module of the present invention;
[0048] Figure 4 This is a timing diagram for merging multi-threaded results in this invention. Detailed Implementation
[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0050] Please see Figure 1 - Figure 4 A text comparison method based on sliding window and dynamically extended KMP algorithm includes the following steps:
[0051] Step 1: Set algorithm parameters and prepare input documents;
[0052] Step 2: Extract substrings from the document to be detected using a sliding window with a step size of 1;
[0053] Step 3: Construct an LPS array containing the original position offsets of valid characters;
[0054] Step 4: Use the improved dynamically expanding KMP algorithm to perform duplicate content matching;
[0055] Step 5: Store the matching results in a hierarchical data structure in real time;
[0056] Step 6: Start multi-threading to perform deduplication, merging, and validation operations on the matching results;
[0057] Step 7: Ensure the accuracy of the merged results through secondary verification;
[0058] Step 8: Output structured text comparison results.
[0059] Specifically, step one involves setting the algorithm parameters and preparing the input document, and the steps are as follows:
[0060] S1.1 First, determine the minimum number of matching characters threshold N. This threshold can be adjusted according to the specific application scenario. For example, N=15 in the education field and N=50 in the bidding field.
[0061] S1.2. Then, simultaneously acquire two input documents to be compared to ensure the integrity of the document data and retain all original character structures in the documents, including spaces, full-width / half-width punctuation, tabs and newlines, and special characters, without performing pre-cleaning processing.
[0062] The advantages are: through the above flexible threshold settings, it can adapt to the document similarity detection needs of different fields, while preserving the original character structure can avoid the modification of the original string by preprocessing, thereby solving the problem of loss of original position information caused by traditional preprocessing methods (such as deleting spaces), laying the foundation for subsequent accurate matching and original position marking.
[0063] Specifically, in step two, a sliding window with a step size of 1 is used to extract the substring of the document to be detected. The specific operation is as follows:
[0064] S2.1. The initial size of the sliding window is set as the minimum number of matching characters threshold N in step one;
[0065] S2.2 Starting from the beginning position of the first document (the document to be detected), scan the document character by character using a step size of 1;
[0066] S2.3. After each move, extract a substring of length N as the pattern string until the end of the window exceeds the document length. During the extraction process, strictly preserve the original character structure of the substring (including spaces and punctuation marks). This process corresponds to... Figure 1 The step of extracting substrings using a sliding window in the overall workflow;
[0067] The advantages are: by using a step size of 1, it ensures that the document is scanned without omission, avoiding the possibility of missing short repetitive segments that may be caused by traditional fixed step sizes (such as N / 2). For example, when there is repetitive content of length N in the document, a step size of 1 can ensure that the window will cover the starting position of the segment, while a scheme with a step size greater than 1 may miss the matching starting point due to skipping scans, thus improving the comprehensiveness of repetitive segment detection.
[0068] Specifically, the process of constructing the LPS array in step three:
[0069] S3.1 For the pattern string (sliding window substring) extracted in step two, construct the LPS array (longest prefix suffix array) required by the KMP algorithm.
[0070] S3.2 During the construction process, the position offset of each valid character (non-special symbols, special symbols include spaces, full-width / half-width punctuation, tabs, newlines and other valid characters) in the original pattern string is recorded synchronously. For example, in the pattern string "a,b;c", the original positions of the valid characters "a", "b", and "c" are 0, 3, and 6 respectively (including the positions of skipped symbols), ensuring that subsequent extended matching can be accurately mapped to the index of the original document.
[0071] The advantages are: by recording the original position offset of valid characters, it ensures that subsequent extended matching can be accurately mapped to the index of the original document, providing data support for accurately marking the true start / end position of the matching segment in the original document, thereby avoiding the disruption of the original matching continuity due to character filtering operations, and ultimately improving the accuracy of the position information of the matching result.
[0072] Specifically, step four uses an improved dynamically expanded KMP algorithm for duplicate content matching. This step is the core matching process. Figure 1The dynamic expansion of KMP matching in the overall workflow consists of three parts:
[0073] (1) Real-time character filtering in the matching stage: In the character comparison step of KMP, when the character to be matched (whether it is a character in the pattern string or the second document (target document)) is a special character, the algorithm does not interrupt the matching process, but directly moves the current character pointer (pattern string pointer or target document pointer) to the next non-special character and continues to perform the comparison. For example, when the pattern string is "a,b" and the corresponding position in the target document is "a+b", the algorithm will skip ",", "+" and spaces, and only compare the consistency of "a" and "b".
[0074] (2) Initial matching judgment: When the pattern string and the target document fragment are compared through real-time character filtering, if the matching length is ≥ the minimum number of matching characters threshold N set in step one, the initial matching is judged to be successful; if the matching fails, the sliding window is moved in the manner of step two to continue to extract new pattern strings for matching.
[0075] (3) Dynamic expansion matching: After the initial matching is successful, the algorithm automatically enters the expansion stage. Starting from the current matching endpoint, the pointers of the pattern string and the target document are moved backward to continue the real-time character filtering mechanism. For each valid character expanded, the original position information of the matching segment (including the position of the skipped symbol) is updated. The expansion terminates when three consecutive valid characters do not match or any document reaches the end, and finally the longest repeating content segment is determined.
[0076] The advantages are: by using a real-time character filtering mechanism, it ensures that symbol differences do not affect the matching of valid content, thereby solving the efficiency bottleneck of the traditional KMP algorithm in text containing complex symbols. For example, in legal document comparison scenarios, the matching speed is improved by more than 50% for text containing a large number of clause numbers and punctuation marks, and no additional preprocessing steps are required; while dynamic extended matching can identify the longest repeated paragraph, thus avoiding fragmented identification of repeated content and ultimately improving the completeness of repeated content detection.
[0077] Specifically, in step five, the matching results are stored in real time to a hierarchical data structure. This hierarchical data structure stores the matching results obtained in step four, and includes a Result class and a MatchResult class, corresponding to... Figure 2 Result storage class diagram.
[0078] Specifically, the Result class stores the core information of a single matching result, including the starting position start1_ in the original document 1 and the starting position start2_ in the original document 2 (accurate to the original character index, including the symbol), the effective character length of the matching segment (excluding skipped symbols) and the original length (including the symbol), and the original string of the matching content (including the symbol, used for subsequent secondary validation).
[0079] Specifically, the MatchResult class stores information by inheriting from the Result class and extending its collection storage capabilities. It records the position information of multiple matching segments through the start1_ (starting position 1) and start2_ (starting position 2) lists, and provides the AddRe method (add duplicate results method) to temporarily merge adjacent matching results, thereby reducing the pressure of subsequent processing.
[0080] The advantages are: by using the hierarchical data structure described above, the matching process and result processing are decoupled, which facilitates the efficient extraction and processing of matching data by the subsequent result merging module. Temporarily merging adjacent matching results also reduces the pressure on subsequent result merging, ultimately improving the efficiency of the entire algorithm process.
[0081] Specifically, in step six, multiple threads are started to perform deduplication, merging, and validation operations on the matching results. This step corresponds to... Figure 3 Flowchart of the result merging module and Figure 4 The multi-threaded result merging sequence diagram includes:
[0082] S4.1 Multi-threaded Parallel Processing Setup: Parallelization is achieved using a producer-consumer model. After the main thread (producer) generates the matching results in step five, it stores the Result object in a thread-safe result queue. An independent merging thread (consumer) extracts the results from the queue and performs subsequent deduplication and merging operations without blocking the main thread's matching process. When the document length exceeds 10,000 characters, multiple merging threads are automatically started (number = number of CPU cores / 2), and dynamic priority scheduling (matching threads have higher priority than merging threads) ensures reasonable resource allocation.
[0083] S4.2 Result Deduplication: Handle inclusion relationships. When the original position range of result A completely covers result B (i.e., A's start1_≤B's start1_ and A's end1_≥B's end1_, the same applies to document 2), then A is retained and B is discarded to avoid redundant records. For example, when both "abcde" and "bcd" are identified as duplicate segments, only "abcde" is retained.
[0084] S4.3 Result Merging: Handling overlapping relationships. When the overlap length of two results accounts for ≥60% of the original length of the shorter result, and the effective character matching rate is ≥95% (after ignoring symbols), they are merged into one record. That is, the smaller value is taken for the starting position and the larger value is taken for the ending position, and the original length is updated to the actual length after merging.
[0085] The advantages are: by using multi-threaded parallel processing, the main process is avoided from being blocked, thereby improving the speed of result processing, especially when processing long documents; the result deduplication and merging operations can eliminate overlapping, contained or duplicate records, and solve the problems of scattered and difficult-to-interpret results in traditional methods. For example, in the comparison of bidding documents, users can directly locate the page number and line number of duplicate clauses in the original document (based on character position conversion), which ultimately greatly improves the efficiency of subsequent processing.
[0086] Specifically, the second verification operation in step seven:
[0087] S5.1. For the merged result in step six, the complete merged segment is re-matched using the improved KMP algorithm (the same real-time character filtering KMP algorithm in step four) to verify whether the merged content is completely consistent after ignoring symbols.
[0088] S5.2 Simultaneously, combined with the sliding window verification mechanism, a sliding window with a step size of 1 is used to cover the merged matching results and calculate the character density (effective character percentage) of the matching segment. When the character density threshold is ≥80%, the merged result is determined to be valid; otherwise, the KMP algorithm is triggered to rescan.
[0089] The advantages are: by using a secondary verification mechanism and a sliding window verification mechanism to jointly form a zero false match guarantee system, it ensures that the merged result has no false matches and completely eliminates accidental matches caused by symbol filtering. For example, in the scenario of financial contract verification, traditional KMP may misjudge contract clauses containing consecutive spaces as matches, while this method completely eliminates such problems through secondary verification, making the false match rate 0.
[0090] Specifically, in step eight, text comparison involves integrating the merged results verified in step seven and outputting structured information, including: the original start / end positions of the merged duplicate segments in both documents (accurate to the character level, including symbol positions), the effective character length of the duplicate segments (used to quantify the duplication ratio), the original text content (including symbols for user verification), and the proportion of the duplicate segments in the documents (effective duplication length / total effective document length). This step corresponds to... Figure 1 The stage in the overall workflow that generates structured output;
[0091] The advantages are: through structured output, it provides users with clear and comprehensive comparison information, enabling users to quickly assess the degree of document duplication, accurately locate the position of duplicate content in the original document, facilitate subsequent modification, verification and other operations, and ultimately improve the readability and usability of the results, meeting the document similarity detection needs of education, publishing, intellectual property and many other fields.
[0092] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A text comparison method based on sliding window and dynamically extended KMP algorithm, characterized in that, Includes the following steps: Step 1: Set algorithm parameters and prepare input documents; Step 2: Extract substrings from the document to be detected using a sliding window with a step size of 1; Step 3: Construct an LPS array containing the original position offsets of valid characters; Step 4: Use the improved dynamically expanding KMP algorithm to perform duplicate content matching; Step 5: Store the matching results in a hierarchical data structure in real time; Step 6: Start multi-threading to perform deduplication, merging, and validation operations on the matching results; Step 7: Ensure the accuracy of the merged results through secondary verification; Step 8: Output structured text comparison results; Step four uses an improved dynamically extended KMP algorithm for duplicate content matching, which specifically includes three parts: (1) Real-time character filtering in the matching stage: In the character comparison step of KMP, when the character to be matched is a special symbol, the algorithm does not interrupt the matching process, but directly moves the current character pointer to the next non-special character and continues to perform the comparison; (2) Initial matching judgment: When the pattern string and the target document fragment are compared through real-time character filtering, if the matching length is ≥ the minimum number of matching characters threshold N set in step one, the initial matching is judged to be successful; if the matching fails, the sliding window is moved in the manner of step two to continue to extract new pattern strings for matching. (3) Dynamic expansion matching: After the initial matching is successful, the algorithm automatically enters the expansion stage. Starting from the current matching endpoint, the pointers of the pattern string and the target document are moved backward to continue the real-time character filtering mechanism. The original position information of the matching segment is updated for each valid character expanded. The expansion terminates when three consecutive valid characters do not match or when any document reaches the end, ultimately determining the longest repeating content segment.
2. The text comparison method based on sliding window and dynamically extended KMP algorithm according to claim 1, characterized in that, The steps in step one, which involve setting the algorithm parameters and preparing the input document, are as follows: S1.1 First, determine the minimum number of matching characters threshold N. This threshold can be adjusted according to the specific application scenario. S1.
2. Then, simultaneously acquire two input documents to be compared, retaining all original character structures in the documents, including spaces, full-width / half-width punctuation, tabs and newlines, and other special characters, without performing pre-cleaning processing.
3. The text comparison method based on sliding window and dynamically extended KMP algorithm according to claim 1, characterized in that, In step two, a sliding window with a step size of 1 is used to extract the substring of the document to be detected. The specific operation is as follows: S2.
1. The initial size of the sliding window is set as the minimum number of matching characters threshold N in step one; S2.
2. Starting from the beginning of the first document, scan the document character by character using a step size of 1; S2.3 After each move, extract a substring of length N as the pattern string until the end of the window exceeds the document length. During the extraction process, strictly preserve the original character structure of the substring.
4. The text comparison method based on sliding window and dynamically extended KMP algorithm according to claim 1, characterized in that, The process of constructing the LPS array in step three: S3.
1. Construct the LPS array required by the KMP algorithm for the pattern string extracted in step two; S3.2 During the construction process, the position offset of each valid character in the original pattern string is recorded synchronously.
5. The text comparison method based on sliding window and dynamically extended KMP algorithm according to claim 1, characterized in that, In step five, the matching results are stored in real time to a hierarchical data structure. The hierarchical data structure is used to store the matching results obtained in step four. This data structure includes a Result class and a MatchResult class.
6. The text comparison method based on sliding window and dynamically extended KMP algorithm according to claim 5, characterized in that, The Result class stores: the core information of a single matching result, including the starting position start1_ in the original document 1 and the starting position start2_ in the document 2, the effective character length and original length of the matching segment, and the original string of the matching content; The MatchResult class stores information by inheriting from the Result class and extending its collection storage capabilities. It records the position information of multiple matching segments through the start1_ and start2_ lists and provides the AddRe method for temporarily merging adjacent matching results.
7. The text comparison method based on sliding window and dynamically extended KMP algorithm according to claim 1, characterized in that, Step six involves starting multiple threads to perform deduplication, merging, and validation operations on the matching results, specifically including: S4.1 Multi-threaded parallel processing settings: Parallelization is achieved using a producer-consumer model. After the main thread generates the matching results in step five, it stores the Result object in a thread-safe result queue. An independent merging thread extracts the results from the queue and performs subsequent deduplication and merging operations without blocking the matching process of the main thread. When the document length exceeds 10,000 characters, multiple merging threads are automatically started and scheduled through dynamic priority. S4.2 Result Deduplication: Handle inclusion relationships. When the original location range of result A completely covers result B, keep A and discard B. S4.3 Result Merging: Handling overlapping relationships. When the overlap length of two results accounts for ≥60% of the original length of the shorter result, and the effective character matching rate is ≥95%, they are merged into one record. That is, the starting position is taken as the smaller value, the ending position is taken as the larger value, and the original length is updated to the actual length after merging.
8. The text comparison method based on sliding window and dynamically extended KMP algorithm according to claim 1, characterized in that, The secondary verification operation in step seven: S5.
1. For the merged result in step six, the complete merged segment is re-matched using the improved KMP algorithm to verify whether the merged content is completely consistent after ignoring symbols. S5.2 Simultaneously, combined with the sliding window verification mechanism, a sliding window with a step size of 1 is used to cover the merged matching results and calculate the character density of the matching fragment. When the character density threshold is ≥80%, the merged result is determined to be valid; otherwise, the KMP algorithm is triggered to rescan.
9. The text comparison method based on sliding window and dynamically extended KMP algorithm according to claim 1, characterized in that, In step eight, text comparison involves integrating the merged results verified in step seven and outputting structured information, including: the original start / end positions of the merged duplicate segments in the two documents, the effective character length of the duplicate segments and the original text content, and the proportion of the duplicate segments in the documents.