Redundant field filtering method based on improved BM algorithm and suffix array

Abstract

The invention discloses a redundant field filtering method based on an improved BM algorithm and a suffix array, and relates to the technical field of pattern recognition and data cleaning, and the method comprises the following steps: S1, obtaining a target text, preprocessing the target text to obtain a standard target text, and obtaining a text character string corresponding to the standard target text, the method comprises the steps of S1, obtaining a text string, S2, obtaining a redundant field text, and obtaining multiple segments of redundant fields corresponding to the redundant field text, and S3, taking the text string as a main string T, constructing a suffix array S based on the main string T, and updating the suffix array S to obtain a new suffix array SA. According to the method, the suffix array and the longest common prefix array are constructed, so that possible matching positions can be quickly positioned in the matching process, unnecessary character comparison operation is avoided, and the number of times of character comparison is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition and data cleaning, and in particular to a redundant field filtering method based on an improved BM algorithm and a suffix array. Background Art

[0002] The BM algorithm (Boyer-Moore algorithm) is an algorithm used to quickly locate pattern strings in string matching. It is an efficient string matching algorithm based on character comparison, using bad character rules and good suffix rules. It is usually faster than traditional brute force matching algorithms.

[0003] Suffix array is a data structure used for string matching and processing. It can convert a string into a set of sorted suffix index arrays, so as to quickly find and process various substrings in the string.

[0004] Traditional methods may need to perform a large number of character comparison operations during the matching process, and there is no effective means to quickly locate possible matching positions, resulting in a large number of character comparisons and low matching efficiency. In addition, traditional methods cannot accurately calculate the maximum safe moving distance when the pattern string moves, and can often only move step by step. They cannot quickly skip areas that cannot be matched, resulting in limited pattern string movement, which reduces the matching speed and accuracy. In addition, due to the large number of character comparisons and limited pattern string movement, traditional methods are prone to incorrect matches or missed matches during string pattern matching, and the matching stability and accuracy are poor. In addition, traditional methods usually do not fully preprocess the target text, such as removing redundant spaces, punctuation marks, etc., and do not perform operations such as stemming and word form restoration, which affects the accuracy and efficiency of subsequent processing. Summary of the invention

[0005] The object of the present invention is to provide a redundant field filtering method based on an improved BM algorithm and a suffix array to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a redundant field filtering method based on an improved BM algorithm and a suffix array, comprising the following steps:

[0007] S1, obtaining a target text, and preprocessing the target text to obtain a standardized target text, and obtaining a text string corresponding to the standardized target text;

[0008] S2. Obtain redundant field text, and obtain multiple redundant fields corresponding to the redundant field text;

[0009] S3, taking the text string as a main string T, constructing a suffix array S based on the main string T, and updating the suffix array S to obtain a new suffix array SA;

[0010] S4, taking the multiple redundant fields as multiple pattern strings P, and calculating a set of positions where the multiple pattern strings P appear in the new suffix array SA;

[0011] S5, obtaining a set of positions where the multi-segment pattern string P appears in the main string T based on the set of positions where the multi-segment pattern string P appears in the new suffix array SA;

[0012] S6. Based on the set of positions where the multiple pattern strings P appear in the main string T, the multiple pattern strings P are deleted from the main string T to obtain a filtering result of the target text.

[0013] Preferably, in step S1, preprocessing the target text includes:

[0014] Remove redundant spaces, tabs, line breaks and blank characters from the target text, and remove punctuation marks from the target text;

[0015] The words in the target text are stemmed and lemmatized to convert them into basic forms of the words.

[0016] Preferably, in step S3, constructing a suffix array S based on the main string T includes the following steps:

[0017] A1. Generate a suffix of the main string T, the suffix includes T[0:], T[1:], T[2:], ..., T[i:], ..., T[n-1:], where n represents the length of the main string T, and T[i:] represents the substring from the i-th character to the end of the main string T;

[0018] A2, initialize an empty array Suffixes to store the suffixes;

[0019] A3, adding the suffix to the empty array Suffixes to obtain the array Suffixes;

[0020] A4. Sort the suffixes in the array Suffixes in lexicographic order by quick sort to obtain a suffix array S.

[0021] Preferably, in step S3, updating the suffix array S to obtain a new suffix array S comprises the following steps:

[0022] B1, read the dictionary window DW and the comparison window CW from the main string T;

[0023] B2. Construct the longest common prefix array LCP based on the suffix array S;

[0024] B3. Use binary search to find the longest common substring LCS in the longest common prefix array LCP, and write the longest common substring LCS in the form of (index, match_len, nextchar) into the triple, where index represents the matching index position, match_len represents the length of the longest common substring LCS, and nextchar represents the next character of index;

[0025] B4. Slide the dictionary window DW and the comparison window CW by match_len characters, update the characters in the dictionary window DW and the comparison window CW, and classify the suffix array S to obtain the compressed suffix array CSA and the uncompressed suffix array UCSA;

[0026] B5. Use binary insertion to insert the elements of the compressed suffix array CSA and the suffix items of the match_len strings at the tail of the dictionary window DW into the uncompressed suffix array UCSA to obtain a new suffix array SA.

[0027] Preferably, in step B2, constructing the longest common prefix array LCP based on the suffix array S includes the following steps:

[0028] C1. Use binary search to find the index position index in the suffix array S that satisfies the condition

[0029] Sx(S[index - 1]) < CW < Sx(S[index]), where Sx(S[index - 1]) represents the substring in the main string T corresponding to the suffix with index - 1 in the suffix array S;

[0030] C2. Compare the characters of the comparison window CW and S[index] one by one. If they are the same, increment the length of the longest common substring LCS;

[0031] C3. Otherwise, determine whether H[index - i + 1] ≥ match_len is satisfied, where H[index - i + 1] represents the length of the longest common prefix of the suffixes sorted as index - i + 1 and index - i in the suffix array S;

[0032] C4. If it is satisfied, record the current length of the longest common substring LCS and repeat step C2. If it is not satisfied, the current longest common substring LCS is the final result.

[0033] Preferably, in step S4, calculating the position set where the multi-segment pattern string P appears in the main string T includes the following steps:

[0034] D1, sequentially dividing the multi-segment pattern string P into P[1, j] and P[j+1, m], where j represents a division node, and m represents the length of the multi-segment pattern string P;

[0035] D2. Perform an exact match between P[1, j] and P[j+1, m] in the suffix array S to obtain a matching result;

[0036] D3. Based on the matching result, obtain a position set L where P[1, j] appears in the new suffix array SA, and obtain a position set R where P[j+1, m] appears in the new suffix array SA.

[0037] Preferably, in step D2, performing an exact match on P[1, j] and P[j+1, m] in the suffix array S to obtain a matching result comprises the following steps:

[0038] E1. Align the starting positions of the suffix array S and P[1, j] and P[j+1, m], and move the current total length of P[1, j] and P[j+1, m];

[0039] E2. Determine whether the value of P[S[m-1+shiftNum]] is 0, where

[0040] P[S[m-1+shiftNum]] represents the character of the multi-segment pattern string P corresponding to the index at position m-1+shiftNum in the suffix array S. If it is 0, compare the other characters in order from left to right and execute step E5;

[0041] E3. If S[m-1+shiftNum] and P[m-1] are not equal or the string matching fails, find the position of the next P[m-1] and record the total moving distance shiftTemp;

[0042] E4. Compare the total moving distance shiftTemp with the moving distances of P[1, j] and P[j+1, m], select the maximum value as the maximum safe distance for the movement of P[1, j] and P[j+1, m], that is, shift=MAX(shiftTemp, the moving distance of P[1, j] and P[j+1, m]), and shiftNum+=shift, and execute step E2;

[0043] E5. Compare the characters of P[1, j] and P[j+1, m] except the character P[m-1] to see if they match the string in the suffix array S. If all characters match, the string matching is successful. Otherwise, execute step E3.

[0044] Preferably, step S5 comprises the following steps:

[0045] S501, traverse the multiple pattern strings P, for each pattern string Pi, use the BM algorithm to find all the positions where Pi appears on the main string T, and record them;

[0046] S502, sorting the multiple pattern strings P in lexicographic order, and then concatenating them to obtain a long pattern string P_all;

[0047] S503, using the suffix array S and the long pattern string P_all, calculate the position set of all pattern strings Pi in the suffix array S;

[0048] S504, according to the position set of all the pattern strings Pi in the suffix array S, by analyzing the longest common prefix array LCP, calculate the position set of each pattern string in the main string T;

[0049] S505, record all the positions where each pattern string Pi appears in the main string T to obtain the final result.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The present invention constructs a suffix array and a longest common prefix array. The method can quickly locate the possible matching position during the matching process, avoid unnecessary character comparison operations, and thus reduce the number of character comparisons. By utilizing the suffix array and the longest common prefix array, the method can accurately calculate the maximum safe distance of each pattern string movement, so that the pattern string can quickly jump over the impossible matching area, thereby increasing the distance of pattern string movement. Moreover, since the number of character comparisons is reduced and the pattern string movement distance is increased, the method can improve the stability and accuracy of string pattern matching. It can more accurately locate the matching position, avoid erroneous matching and missed matching, thereby improving the matching accuracy.

[0052] 2. The present invention preprocesses the target text in step S1, including removing redundant spaces, tabs, line breaks and blank characters, removing punctuation marks, and stemming and restoring words. These preprocessing steps can make subsequent processing more accurate and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of the overall method flow provided by an embodiment of the present invention;

[0054] Figure 2 A flowchart of constructing a suffix array S provided in an embodiment of the present invention;

[0055] Figure 3 A flowchart of updating the suffix array S provided by an embodiment of the present invention;

[0056] Figure 4 A flowchart of constructing the longest common prefix array LCP provided by an embodiment of the present invention;

[0057] Figure 5 A flowchart of calculating a set of positions where a multi-segment pattern string P appears in a main string T provided by an embodiment of the present invention;

[0058] Figure 6 A flowchart for matching P[1, j] and P[j+1, m] in a suffix array S provided by an embodiment of the present invention;

[0059] Figure 7 This is a flowchart of step S5 provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] See also Figure 1-7 The present invention provides a technical solution: a redundant field filtering method based on an improved BM algorithm and a suffix array, comprising the following steps:

[0062] S1, obtaining a target text, and preprocessing the target text to obtain a standardized target text, and obtaining a text string corresponding to the standardized target text;

[0063] S2. Obtain redundant field text, and obtain multiple redundant fields corresponding to the redundant field text;

[0064] S3, taking the text string as a main string T, constructing a suffix array S based on the main string T, and updating the suffix array S to obtain a new suffix array SA;

[0065] S4, taking the multiple redundant fields as multiple pattern strings P, and calculating a set of positions where the multiple pattern strings P appear in the new suffix array SA;

[0066] S5, obtaining a set of positions where the multi-segment pattern string P appears in the main string T based on the set of positions where the multi-segment pattern string P appears in the new suffix array SA;

[0067] S6. Based on the set of positions where the multiple pattern strings P appear in the main string T, the multiple pattern strings P are deleted from the main string T to obtain a filtering result of the target text.

[0068] In the step S1, the target text is preprocessed, including:

[0069] Remove redundant spaces, tabs, line breaks and blank characters from the target text, and remove punctuation marks from the target text;

[0070] Stemming and lemmatization are performed on the words in the target text to convert them into the basic forms of the words;

[0071] In step S3, constructing a suffix array S based on the main string T includes the following steps:

[0072] A1. Generate a suffix of the main string T, the suffix includes T[0:], T[1:], T[2:], ..., T[i:], ..., T[n-1:], where n represents the length of the main string T, and T[i:] represents the substring from the i-th character to the end of the main string T;

[0073] A2, initialize an empty array Suffixes to store the suffixes;

[0074] A3, adding the suffix to the empty array Suffixes to obtain the array Suffixes;

[0075] A4. Sort the suffixes in the array Suffixes in lexicographic order by quick sort to obtain a suffix array S;

[0076] In step S3, the suffix array S is updated to obtain a new suffix array S, including the following steps:

[0077] B1, read the dictionary window DW and the comparison window CW from the main string T;

[0078] B2. constructing a longest common prefix array LCP based on the suffix array S;

[0079] B3. Use binary search to find the longest common substring LCS in the longest common prefix array LCP, and write the longest common substring LCS in the form of (index, match_len, nextchar) into a triple, where index represents the matching index position, match_len represents the length of the longest common substring LCS, and nextchar represents the next character of index;

[0080] B4. Slide the dictionary window DW and the comparison window CW by match_len characters, update the characters in the dictionary window DW and the comparison window CW, and classify the suffix array S to obtain a compressed suffix array CSA and an uncompressed suffix array UCSA;

[0081] B5. Use binary insertion to insert the elements of the compressed suffix array CSA and the suffix items of the match_len string at the tail of the dictionary window DW into the uncompressed suffix array UCSA to obtain a new suffix array SA;

[0082] In step B2, constructing the longest common prefix array LCP based on the suffix array S includes the following steps:

[0083] C1. Use binary search to find the index position index in the suffix array S that satisfies the condition

[0084] Sx(S[index - 1]) < CW < Sx(S[index]), where Sx(S[index - 1]) represents the substring in the main string T corresponding to the suffix with index - 1 in the suffix array S;

[0085] In this step, binary search is an algorithm for finding a specific element in an ordered array. Its basic idea is to divide the search interval into two parts each time. By comparing the relationship between the target value and the middle element, the interval where the target value may exist can be determined, and then the search range is continuously narrowed until the target value is found or it is determined that the target value does not exist;

[0086] C2. Compare the characters of the comparison window CW and S[index] one by one. If they are the same, increment the length of the longest common substring LCS;

[0087] C3. Otherwise, determine whether H[index - i + 1] ≥ match_len is satisfied, where H[index - i + 1] represents the length of the longest common prefix of the suffixes sorted as index - i + 1 and index - i in the suffix array S;

[0088] C4, if it is satisfied, record the current length of the longest common substring LCS and repeat step C2; if it is not satisfied, the current longest common substring LCS is the final result;

[0089] In step S4, calculating the position set where the multi-segment pattern string P appears in the main string T includes the following steps:

[0090] D1, sequentially dividing the multi-segment pattern string P into P[1, j] and P[j+1, m], where j represents a division node, and m represents the length of the multi-segment pattern string P;

[0091] D2. Perform an exact match between P[1, j] and P[j+1, m] in the suffix array S to obtain a matching result;

[0092] D3. Based on the matching result, obtain a set L of positions where P[1, j] appears in the new suffix array SA, and obtain a set R of positions where P[j+1, m] appears in the new suffix array SA;

[0093] In the step D2, the P[1, j] and the P[j+1, m] are accurately matched in the suffix array S to obtain a matching result, including the following steps:

[0094] E1. Align the starting positions of the suffix array S and P[1, j] and P[j+1, m], and move the current total length of P[1, j] and P[j+1, m];

[0095] E2. Determine whether the value of P[S[m-1+shiftNum]] is 0, where

[0096] P[S[m-1+shiftNum]] represents the character of the multi-segment pattern string P corresponding to the index at position m-1+shiftNum in the suffix array S. If it is 0, compare the other characters in order from left to right and execute step E5;

[0097] E3. If S[m-1+shiftNum] and P[m-1] are not equal or the string matching fails, find the position of the next P[m-1] and record the total moving distance shiftTemp;

[0098] E4. Compare the total moving distance shiftTemp with the moving distances of P[1, j] and P[j+1, m], select the maximum value as the maximum safe distance for the movement of P[1, j] and P[j+1, m], that is, shift=MAX(shiftTemp, the moving distance of P[1, j] and P[j+1, m]), and shiftNum+=shift, and execute step E2;

[0099] E5. Compare the characters of P[1, j] and P[j+1, m] except the character P[m-1] to see if they match the string in the suffix array S. If all characters match, the string matching is successful. Otherwise, execute step E3.

[0100] The step S5 comprises the following steps:

[0101] S501, traverse the multiple pattern strings P, for each pattern string Pi, use the BM algorithm to find all the positions where Pi appears on the main string T, and record them;

[0102] S502, sorting the multiple pattern strings P in lexicographic order, and then concatenating them to obtain a long pattern string P_all;

[0103] S503, using the suffix array S and the long pattern string P_all, calculate the position set of all pattern strings Pi in the suffix array S;

[0104] In the step, two algorithms can be used: one is the binary search algorithm, and the other is to use the LCP array of SA and P_all for matching;

[0105] S504, according to the position set of all the pattern strings Pi in the suffix array S, by analyzing the longest common prefix array LCP, calculate the position set of each pattern string in the main string T;

[0106] In this step, two algorithms can be used: one is to trace back the suffix link pointer of SA, and the other is to calculate based on the LCP array and the height array obtained from SA

[0107] S505, record all the positions where each pattern string Pi appears in the main string T to obtain the final result.

[0108] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0109] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A redundant field filtering method based on an improved BM algorithm and a suffix array, characterized in that: The following steps are involved: S1, obtaining a target text, and preprocessing the target text to obtain a standardized target text, and obtaining a text string corresponding to the standardized target text; S2. Obtain redundant field text, and obtain multiple redundant fields corresponding to the redundant field text; S3, taking the text string as a main string T, constructing a suffix array S based on the main string T, and updating the suffix array S to obtain a new suffix array SA; S4, taking the multiple redundant fields as multiple pattern strings P, and calculating a set of positions where the multiple pattern strings P appear in the new suffix array SA; S5, obtaining a set of positions where the multi-segment pattern string P appears in the main string T based on the set of positions where the multi-segment pattern string P appears in the new suffix array SA; S6. Based on the set of positions where the multiple pattern strings P appear in the main string T, the multiple pattern strings P are deleted from the main string T to obtain a filtering result of the target text.

2. According to claim 1, a redundant field filtering method based on improved BM algorithm and suffix array is characterized in that: In the step S1, the target text is preprocessed, including: Remove redundant spaces, tabs, line breaks and blank characters from the target text, and remove punctuation marks from the target text; The words in the target text are stemmed and lemmatized to convert them into basic forms of the words.

3. The redundant field filtering method based on the improved BM algorithm and suffix array according to claim 1, characterized in that: In step S3, constructing a suffix array S based on the main string T includes the following steps: A1. Generate a suffix of the main string T, the suffix includes T[0:], T[1:], T[2:], ..., T[i:], ..., T[n-1:], where n represents the length of the main string T, and T[i:] represents the substring from the i-th character to the end of the main string T; A2, initialize an empty array Suffixes to store the suffixes; A3, adding the suffix to the empty array Suffixes to obtain the array Suffixes; A4. Sort the suffixes in the array Suffixes in lexicographic order by quick sort to obtain a suffix array S.

4. The redundant field filtering method based on the improved BM algorithm and suffix array according to claim 3 is characterized in that: In step S3, the suffix array S is updated to obtain a new suffix array S, including the following steps: B1, read the dictionary window DW and the comparison window CW from the main string T; B2. constructing a longest common prefix array LCP based on the suffix array S; B3, using binary search to find the longest common substring LCS in the longest common prefix array LCP, and write the longest common substring LCS into a triple in the form of (index, match_len, nextchar), where index represents the matching index position, match_len represents the length of the longest common substring LCS, and nextchar represents the next character of index; B4, the dictionary window DW and the comparison window CW slide match_len characters, update the characters in the dictionary window DW and the comparison window CW, and classify the suffix array S to obtain a compressed suffix array CSA and an uncompressed suffix array UCSA; B5. Use binary insertion to insert the elements of the compressed suffix array CSA and the suffix items of the match_len string at the tail of the dictionary window DW into the uncompressed suffix array UCSA to obtain a new suffix array SA.

5. The redundant field filtering method based on the improved BM algorithm and suffix array according to claim 4 is characterized in that: In step B2, constructing the longest common prefix array LCP based on the suffix array S includes the following steps: C1. Use binary search to find the index position index in the suffix array S that satisfies the condition Sx(S[index - 1]) < CW < Sx(S[index]), where Sx(S[index - 1]) represents the substring in the main string T corresponding to the suffix with index index - 1 in the suffix array S; C2. Compare the characters of the comparison window CW and S[index] one by one. If they are the same, increment the length of the longest common substring LCS; C3. Otherwise, determine whether H[index - i + 1] ≥ match_len is satisfied, where H[index - i + 1] represents the length of the longest common prefix of the suffixes sorted as index - i + 1 and index - i in the suffix array S; C4. If it is satisfied, record the current length of the longest common substring LCS and repeat step C2. If it is not satisfied, the current longest common substring LCS is the final result.

6. The redundant field filtering method based on the improved BM algorithm and suffix array according to claim 1, characterized in that: In step S4, calculating the set of positions where the multi-segment pattern string P appears in the main string T includes the following steps: D1. Sequentially divide the multi-segment pattern string P into P[1, j] and P[j + 1, m], where j represents the division node and m represents the length of the multi-segment pattern string P; D2. Perform exact matching on P[1, j] and P[j + 1, m] in the suffix array S to obtain a matching result; D3. Based on the matching result, obtain the set of positions L where P[1, j] appears in the new suffix array SA, and obtain the set of positions R where P[j + 1, m] appears in the new suffix array SA.

7. The redundant field filtering method based on improved BM algorithm and suffix array according to claim 6, characterized in that: In step D2, performing exact matching on P[1, j] and P[j + 1, m] in the suffix array S to obtain a matching result includes the following steps: E1. Align the starting positions of the suffix array S, P[1, j], and P[j + 1, m], and move P[1, j] and P[j + 1, m] by the current total length; E2. Determine whether the value of P[S[m - 1 + shiftNum]] is 0, where P[S[m - 1 + shiftNum]] represents the character of the multi-segment pattern string P corresponding to the index at position m - 1 + shiftNum in the suffix array S. If it is 0, compare the other characters in order from left to right and execute step E5; E3. If S[m - 1 + shiftNum] and P[m - 1] are not equal or the string matching fails, find the next position of P[m - 1] and record the total movement distance shiftTemp; E4. Compare the total moving distance shiftTemp with the moving distances of P[1, j] and P[j+1, m], select the maximum value as the maximum safe distance for the movement of P[1, j] and P[j+1, m], that is, shift=MAX(shiftTemp, the moving distance of P[1, j] and P[j+1, m]), and shiftNum+=shift, and execute step E2; E5. Compare the characters of P[1, j] and P[j+1, m] except the character P[m-1] to see if they match the string in the suffix array S. If all characters match, the string matching is successful. Otherwise, execute step E3.

8. The redundant field filtering method based on the improved BM algorithm and suffix array according to claim 4 is characterized in that: The step S5 comprises the following steps: S501, traverse the multiple pattern strings P, for each pattern string Pi, use the BM algorithm to find all the positions where Pi appears on the main string T, and record them; S502, sorting the multiple pattern strings P in lexicographic order, and then concatenating them to obtain a long pattern string P_all; S503, using the suffix array S and the long pattern string P_all, calculate the position set of all pattern strings Pi in the suffix array S; S504, according to the position set of all the pattern strings Pi in the suffix array S, by analyzing the longest common prefix array LCP, calculate the position set of each pattern string in the main string T; S505, record all the positions where each pattern string Pi appears in the main string T to obtain the final result.

Images