Standard text review method and device, equipment and medium

Abstract

The invention provides a standard text review method and device, equipment and a medium. The method comprises the steps that a pattern string set is acquired; the pattern string set comprises a plurality of pattern strings; constructing a jump table and a hash table according to suffix character blocks of all pattern string matching windows, constructing a prefix table according to prefix character blocks of the matching windows, and constructing a displacement table according to non-suffix character blocks of the matching windows; obtaining a combined character block, and constructing a Bloom filter according to the combined character block; the combined character block is composed of a suffix character block of the current matching window and a prefix character block of the next matching window; and adopting the hash table, the jump table, the prefix table, the displacement table and the Bloom filter to perform mobile scanning matching on the standard text through the to-be-verified matching window to obtain a text review result. According to the scheme, the situation that accurate matching is not needed can be avoided, so that the matching process is accelerated, meanwhile, the method has the advantages of being large in jumping distance and small in accurate verification frequency, and the text review efficiency is further improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a standard text review method, device, equipment and medium. Background Art

[0002] With the development of information technology and the explosive growth of Internet content, text information has become an important way for people to communicate, learn and acquire knowledge. Among them, standardization documents are "documents formulated through standardization activities". Standardization documents are a type of normative documents. The main difference between them and other normative documents in the entropy of the formation process is that they are generated by standardization activities. In a large amount of standard text data, there may be various quality problems, such as grammatical errors, spelling errors, logical incomprehension, semantic ambiguity and other problems. In order to ensure the accuracy and consistency of standard documents, it is particularly important to review the standard text.

[0003] At present, in the process of reviewing standard texts, the relevant technology uses the Wu-Manber multi-pattern matching algorithm to detect specific words or phrases in the text. However, this solution has some limitations, such as small jump distance, large hash conflict, and it takes a long time to perform accurate verification, which reduces the efficiency of text review. Summary of the invention

[0004] In view of this, the present invention provides a standard text review method, device, equipment and medium, which at least partially solve the problems existing in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a standard text review method, the method comprising:

[0006] Acquire a pattern string set; the pattern string set includes multiple pattern strings;

[0007] Constructing a jump table and a hash table according to the suffix character blocks of all the pattern string matching windows, constructing a prefix table according to the prefix character blocks of the matching windows, and constructing a displacement table according to the non-suffix character blocks of the matching windows;

[0008] Acquire a combined character block, and construct a Bloom filter according to the combined character block; the combined character block is composed of a suffix character block of a current matching window and a prefix character block of a next matching window;

[0009] The hash table, the jump table, the prefix table, the displacement table and the Bloom filter are used to perform mobile scanning matching on the standard text through the matching window to be verified to obtain a text review result.

[0010] Optionally, constructing a displacement table according to the non-suffix character blocks of the matching window includes:

[0011] Encode the pattern string to determine the number of pattern strings in the pattern string set and the length value of the shortest pattern string in the pattern string; the pattern string includes a suffix character block and a non-suffix character block;

[0012] Calculate the character block size value according to the number of pattern strings and the shortest pattern string length value;

[0013] Based on the character block size value, calculating the non-suffix character block and a first jump value corresponding to the non-suffix character block;

[0014] The non-suffix character block and the first jump value corresponding to the non-suffix character block are stored correspondingly to construct the displacement table, and the first jump value corresponding to the suffix character block in the displacement table is zero.

[0015] Optionally, a jump table is constructed according to the suffix character blocks of all the pattern string matching windows, including:

[0016] When the suffix character block is in the first situation, the second jump value is m-B+4; the first situation is that the suffix character block does not appear in the matching window of other pattern strings; the suffix character block is the character block currently being verified that is located at the rightmost end of the matching window and has a character block size of B; m is the shortest pattern string length value, and B is the character block size value;

[0017] When the suffix character block is in the second situation, the second jump value is mB; the second situation is that the suffix character block only appears at the leftmost end of other pattern strings;

[0018] When the suffix character block is in other situations, the second jump value is 4; the other situations are situations other than the first situation and the second situation;

[0019] The jump table is constructed according to the second jump value and the suffix character block.

[0020] Optionally, the hash table, the jump table, the prefix table, the displacement table and the Bloom filter are used to perform mobile scanning matching on the standard text through the matching window to be checked to obtain a text review result, including:

[0021] Obtain the target character block of the matching window to be verified and the first jump value of the target character block in the displacement table;

[0022] Determining whether the first jump value is greater than zero;

[0023] When the first jump value is not greater than zero, the suffix character block of the matching window and the prefix character block of the next matching window are used to construct a target combined character block;

[0024] Using the Bloom filter to verify the target combined character block in the standard text to obtain a verification result; the verification result is used to indicate a verification failure or a verification success;

[0025] According to the verification result, the prefix table, the hash table and the jump table are used to scan and match the standard text to obtain the text review result.

[0026] Optionally, after determining whether the first jump value is greater than zero, the method further includes:

[0027] When the first jump value is greater than zero, skipping the first jump value positions;

[0028] Continue searching and verifying the standard text until the standard text is verified and the text review result is obtained.

[0029] Optionally, according to the verification result, the prefix table, the hash table and the jump table are used to scan and match the standard text to obtain the text review result, including:

[0030] When the verification result indicates that the verification is successful, the target character block in the matching window is verified based on the prefix table and the hash table to obtain the text review result;

[0031] When the verification result indicates that the verification has failed, the standard text is scanned and matched based on the jump table to obtain the text review result.

[0032] Optionally, scanning and matching the standard text based on the jump table to obtain the text review result includes:

[0033] Obtain a second jump value of the target character block in the jump table;

[0034] Jump to the second jump value position, continue to search and verify from the standard text, until the standard text verification is completed, and obtain the text review result.

[0035] In a second aspect, the present application provides a standard text review device, the device comprising:

[0036] An acquisition module, used to acquire a pattern string set; the pattern string set includes multiple pattern strings;

[0037] A first construction module is used to construct a jump table and a hash table according to the suffix character blocks of all the pattern string matching windows, to construct a prefix table according to the prefix character blocks of the matching windows, and to construct a displacement table according to the non-suffix character blocks of the matching windows;

[0038] A second construction module is used to obtain a combined character block and construct a Bloom filter according to the combined character block; the combined character block is composed of a suffix character block of a current matching window and a prefix character block of a next matching window;

[0039] The review module is used to use the hash table, the jump table, the prefix table, the displacement table and the Bloom filter to perform mobile scanning matching on the standard text through the matching window to be verified to obtain a text review result.

[0040] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the standard text review method of the first aspect described above is implemented.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to implement the standard text review method as described in the first aspect above.

[0042] The standard text review method, apparatus, device and medium provided in the embodiments of the present application obtain a pattern string set, which includes multiple pattern strings; construct a jump table and a hash table based on the suffix character blocks of all pattern string matching windows, and construct a prefix table based on the prefix character blocks of the matching window, and construct a displacement table based on the non-suffix character blocks of the matching window; obtain a combined character block, and construct a Bloom filter based on the combined character block; the combined character block is composed of the suffix character block of the current matching window and the prefix character block of the next matching window; use the hash table, jump table, prefix table, displacement table and Bloom filter to perform mobile scanning matching on the standard text through the matching window to be verified, and obtain the text review result. Compared with the prior art, this technical solution can dynamically perform matching jump processing according to the data in the displacement table by accurately constructing a displacement table according to the non-suffix character blocks of the matching window, thereby increasing the jump distance and increasing the jump amplitude, and by constructing a jump table according to the suffix character blocks of all pattern string matching windows, the jump value can be set in advance, and the time consumption of the matching stage can be transferred to the preprocessing stage, thereby speeding up the matching performance, and by accurately constructing a Bloom filter based on the combined character blocks, the character blocks can be preliminarily filtered to avoid the situation where precise matching is not required, thereby speeding up the matching process, and when the shortest pattern string is long and the same prefix and suffix exist in each pattern string, it has the advantages of a large jump distance and a small number of precise verification times, thereby further improving the efficiency of text review. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 Schematic diagram of the process flow of the standard text review method provided in the embodiment of the present application

[0045] Figure 2 A flow chart of a method for constructing a displacement table according to non-suffix character blocks in a matching window provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a method for scanning and matching a standard text to obtain a text review result provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of constructing a hash table, a displacement table, a jump table, a prefix table and a Bloom filter provided in an embodiment of the present application;

[0048] Figure 5 A pseudo code diagram of the scan matching stage provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structure of the displacement table, jump table and Bloom filter constructed in the embodiment of the present application;

[0050] Figure 7 A schematic diagram of the process of performing scanning and matching on a standard text T provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the process flow of a standard text review device provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0054] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0055] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0056] At present, in the process of reviewing standard texts, the related technology uses the Wu-Manber multi-pattern matching algorithm to detect specific words or phrases in the text. However, this solution has some limitations, such as small jump distance, large hash collision, and it takes a long time to accurately check, which reduces the efficiency of text review. Among them, the Wu-Manber algorithm is mainly used to find multiple patterns (i.e., multiple substrings) in a text at the same time, and can complete the matching task within the time complexity of O(n), where n is the length of the text.

[0057] Based on the above-mentioned defects, the present application provides a standard text review method, device, equipment and medium. Compared with the prior art, the technical solution accurately constructs a displacement table according to the non-suffix character blocks of the matching window, and can dynamically perform matching jump processing according to the data in the displacement table, thereby improving the jump distance and increasing the jump amplitude. By constructing a jump table according to the suffix character blocks of all pattern string matching windows, the jump value can be set in advance, and the time consumption of the matching stage is transferred to the preprocessing stage, thereby speeding up the matching performance. By accurately constructing a Bloom filter based on the combined character blocks, the character blocks can be preliminarily filtered, avoiding the situation where precise matching is not required, thereby speeding up the matching process. When the shortest pattern string is long and the same prefix and suffix exist in each pattern string, it has the advantages of a large jump distance and a small number of precise verification times, thereby further improving the text review efficiency.

[0058] For ease of understanding and explanation, the following Figures 1 to 9 The standard text review method, device, equipment, and medium provided in the embodiments of the present application are described in detail.

[0059] Figure 1 The figure shows a flow chart of a standard text review method according to an embodiment of the present application, which can be executed by a computer device. Figure 1 As shown, the method includes:

[0060] S101. Obtain a pattern string set; the pattern string set includes multiple pattern strings.

[0061] It should be noted that the above pattern string set can be the Chinese text to be verified, which can include multiple pattern strings, and the character blocks corresponding to each pattern string can be the same or different, and the Chinese text can be in PDF format. For example, the above pattern string set can be P = {incredible, indescribable, information age}, then "incredible", "indescribable", "information age" are the pattern strings respectively.

[0062] The above-mentioned pattern string set can be obtained by importing from an external device, or obtained from a database, or obtained from a blockchain. The embodiment of the present application does not impose any limitation on the method of obtaining the pattern string set.

[0063] S102, constructing a jump table and a hash table according to the suffix character blocks of all pattern string matching windows, constructing a prefix table according to the prefix character blocks of the matching windows, and constructing a displacement table according to the non-suffix character blocks of the matching windows.

[0064] It is understandable that the above-mentioned jump table can be called Jump Table, which is a data structure used to speed up the matching process. It provides different jump distances based on the target character block H. The target character block usually refers to the character block of size B on the far right of the matching window. The displacement table can be called Shift Table, which is a key part of the original Wu-Manber algorithm and is used to determine how many characters the window should slide forward when the target character block does not completely match any pattern. The prefix table can be called Prefix Table, which is used to quickly determine whether a complete pattern exists in the precise verification stage. Once a possible match is found, this table is used to reduce unnecessary character comparisons.

[0065] The prefix table and hash table are used in the exact matching process after the Bloom filter is verified to ensure that the standard text does contain the complete pattern. The displacement table and jump table determine the distance that should be skipped when the target character block does not match.

[0066] In one embodiment, the present application provides a specific implementation of a method for constructing a displacement table based on a non-suffix character block in a matching window. Figure 2 As shown, the method includes:

[0067] S201. Encode the pattern string to determine the number of pattern strings in the pattern string set and the length of the shortest pattern string in the pattern string; the pattern string includes a suffix character block and a non-suffix character block.

[0068] S202. Calculate the character block size value according to the number of pattern strings and the shortest pattern string length value.

[0069] S203. Calculate the non-suffix character block and the first jump value corresponding to the non-suffix character block based on the character block size value.

[0070] S204. Store the non-suffix character block and the first jump value corresponding to the non-suffix character block in correspondence to construct a displacement table, in which the first jump value corresponding to the suffix character block in the displacement table is zero.

[0071] It is understandable that, since the PDF Chinese text encoding has the characteristic that four characters constitute one Chinese character, in the preprocessing stage, when initializing the SHIFT displacement table, the key value can be four units apart, and when initializing the SHIFT hash table, the key value should be four units apart, so as to ensure that each jump spans a complete Chinese character. The displacement table includes a character block and a corresponding first jump value, and the first jump value is used to characterize the jump distance. Among them, if the character block of the current window does not appear in any pattern string, the default jump value is changed from m-B+1 to m-B+4. This is because each Chinese character occupies four characters, so the minimum effective jump distance should be four characters. In the matching stage, the first jump value after the precise verification result is fixed to 1 in the Wu-Manber algorithm, but the first jump value can be obtained dynamically, and due to the particularity of CID encoding, the minimum jump value is 4.

[0072] Specifically, after obtaining multiple pattern strings, a preset encoding algorithm can be used to encode them to obtain the encoded result, and then the non-suffix character block and the suffix character block of the matching window are determined according to the encoded result, the first jump value of the suffix character block is set to zero, and the number of pattern strings in the pattern string set and the shortest pattern string length value in the pattern string are determined, and then the character block size value is calculated by the following formula:

[0073] B=[4×logΣ(2×m×r)];

[0074] Among them, B is the character block size value, Σ is the size of the pattern string set, m is the length value of the shortest pattern string, and r is the number of pattern strings.

[0075] After determining the character block size value, the non-suffix character block and the first jump value corresponding to the non-suffix character block can be calculated, wherein the first jump value of each character block is a multiple of 4, thereby increasing the jump range. For example, if a character block does not appear in any pattern string, its jump value is m-B+4.

[0076] Exemplarily, assuming that the pattern string set is P = {incredible, wonderful, information age}, it can be encoded first, such as CID encoding, and the result obtained after encoding is P = {4e0d53ef601d8bae, 59994e0d53ef8a00, 4fe1606f65f64ee3}. According to the result, the number of pattern strings is determined to be 3, and the shortest pattern string length is 16. The character block size value can be determined based on the number of pattern strings and the shortest pattern string length. Assuming that the character block size value B = 8, the SHIFT displacement table constructed according to the pattern string set is shown in Table 1 below, which includes each character block and the corresponding jump value:

[0077] Table 1

[0078]

[0079] It can be seen from Table 1 above that the jump value corresponding to each character block is a multiple of 4, which increases the jump distance and thus increases the jump amplitude.

[0080] In this embodiment, by setting the first jump value to a multiple of 4, the characteristics of Chinese text can be better utilized, the jump range is greatly increased, the matching efficiency is improved, and unnecessary comparison operations are reduced.

[0081] In another embodiment of the present application, the present application provides a specific implementation method for constructing a jump table based on the suffix character blocks of all pattern string matching windows. The method includes:

[0082] When the suffix character block is in the first situation, the second jump value is m-B+4; the first situation is that the target character block does not appear in the matching window of other pattern strings; the suffix character block is the character block currently being verified that is located at the rightmost end of the matching window and has a character block size of B; m is the shortest pattern string length value, and B is the character block size value; when the target character block is in the second situation, the second jump value is mB; the second situation is that the suffix character block only appears at the leftmost end of other pattern strings; when the suffix character block is in other situations, the second jump value is 4; other situations are situations other than the first and second situations; a jump table is constructed based on the second jump value and the suffix character block.

[0083] It should be noted that in the Wu-Manber algorithm, the character block of size B at the rightmost end of the matching window can be used as the suffix string and named H. When shift[H]=0, an exact check will be performed after checking the prefix table, and the matching window will only slide to the right by a distance of 1 after the check. When the standard text is large in scale, the number of exact checks is large and the sliding distance is only 1, resulting in a large number of matches, which leads to low matching efficiency of the algorithm. In order to solve this problem, a jump table is constructed to improve the matching efficiency. In the jump table, its key is the same as that of the hash table, and the corresponding second jump value represents the jump value after the exact match. The jump table includes character blocks and corresponding second jump values, or it may include hash values ​​of character blocks and corresponding second jump values.

[0084] In the process of building the jump table, when the suffix character block H to be verified does not appear in the matching window of other pattern strings, the corresponding second jump value jump[H] = m-B+4. The purpose of this is to jump as far as possible to those parts that may not contain matching items; when it only appears at the leftmost end of other pattern strings, the corresponding second jump value jump[H] = mB, because if H is only at the beginning of the pattern, then we can safely skip the length of the entire pattern minus B; for other cases, the corresponding second jump value jump[H] = 4. This value is an empirical value that aims to balance the jump distance and avoid the risk of excessive jumps.

[0085] In this embodiment, since a jump table including a second jump value is constructed in the preprocessing stage, the second jump value can be directly obtained from the jump table after the precise verification in the matching stage is completed, thereby transferring the unnecessary time consumption in the matching stage to the preprocessing stage, accelerating the matching speed and improving the matching efficiency.

[0086] S103, obtaining a combined character block, and constructing a Bloom filter according to the combined character block; the combined character block is composed of a suffix character block of the current matching window and a prefix character block of the next matching window.

[0087] It should be noted that the above-mentioned Bloom filter can be called a bloom filter, which is a probabilistic data structure used to quickly determine whether an element may exist in a set. Bloom filter is an efficient random data storage structure proposed by Bloom in 1970. It uses a bitmap to concisely represent a set and quickly determine whether an element belongs to the set. Bloom filter is a data structure based on multi-hash function mapping compressed parameter space, mapping the values ​​after k hash function operations into a bitmap, with high space and time efficiency. In this application, the pre-constructed Bloom filter is used to perform preliminary screening of the combined character block HB to reduce unnecessary precise verification.

[0088] The above-mentioned combined character block is a new character block HB with a length of B, which is composed of the suffix character block of the current matching window and the prefix character block of the next matching window. In the preprocessing stage, all combined character blocks HB that are not the shortest pattern strings are added to the Bloom filter. In the matching stage, when a possible match is detected, it is calculated whether the combined character block HB is in the Bloom filter before searching the prefix table and the hash table. If it exists, an accurate check is performed, otherwise it is directly jumped according to the second jump value jump[H] in the jump table to find out whether there is a match, thereby obtaining the text review result.

[0089] The use of Bloom filters in this embodiment can greatly reduce the number of situations that require accurate verification, thereby improving efficiency. In addition, a more flexible jump strategy is provided through the jump table, avoiding the situation where only one unit is moved each time, especially when the target character block H does not appear in any pattern. In actual implementation, this improvement enables the Wu-Manber algorithm to more efficiently handle the matching problem of large-scale text and a large number of keywords.

[0090] S104, using a hash table, a jump table, a prefix table, a displacement table and a Bloom filter, performing a moving scan match on the standard text through the matching window to be verified to obtain a text review result.

[0091] It should be noted that the above standard text is a pre-edited template text that meets the preset standards. The standard text may have an editing function. The preset standards may include national standards, industry standards, group standards, enterprise standards, local standards, and standardization guidance technical documents. The text review result is used to indicate whether the matching window is in the standard text.

[0092] In one embodiment, the present application provides a specific implementation method of using a hash table, a jump table, a prefix table, a displacement table and a Bloom filter to perform mobile scanning matching on the standard text through the matching window to be checked to obtain the text review result, see Figure 3 As shown, the method includes:

[0093] S301, obtaining the target character block of the matching window to be verified and the first jump value of the target character block in the displacement table.

[0094] S302: Determine whether the first jump value is greater than zero.

[0095] S303: When the first jump value is not greater than zero, the suffix character block of the matching window and the prefix character block of the next matching window are used to construct a target combined character block.

[0096] S304, using a Bloom filter to verify the target combined character block in the standard text to obtain a verification result; the verification result is used to indicate a verification failure or a verification success.

[0097] S305. According to the verification result, the prefix table, hash table and jump table are used to scan and match the standard text to obtain the text review result.

[0098] Specifically, after obtaining the pattern string set, the shortest pattern string length value can be determined, and the suffix character blocks of all pattern string matching windows can be used to build a jump table and a hash table, and a prefix table can be built based on the prefix character blocks of the matching window, and a displacement table can be built based on the non-suffix character blocks of the matching window, and finally a Bloom filter can be built based on the combined character block HB.

[0099] Optionally, after determining whether the first jump value is greater than zero, the method further includes:

[0100] When the first jump value is greater than zero, the first jump value position is skipped; the search and verification are continued from the standard text until the standard text verification is completed to obtain the text review result.

[0101] In the process of using prefix tables, hash tables and jump tables to scan and match standard texts to obtain text review results, specifically when the verification result indicates that the verification is successful, the target character block in the matching window is verified based on the prefix table and the hash table to obtain the text review result; when the verification result indicates that the verification fails, the standard text is scanned and matched based on the jump table to obtain the text review result.

[0102] Among them, the standard text is scanned and matched based on the jump table to obtain the text review results, including:

[0103] Get the second jump value of the target character block in the jump table; jump to the second jump value position, continue to search and verify from the standard text, until the standard text verification is completed, and obtain the text review result.

[0104] See also Figure 4 As shown, Figure 4A schematic diagram of a hash table, a jump table, a prefix table, a shift table and a bloom filter constructed in the preprocessing stage provided in an embodiment of the present application, wherein the gray portion represents a matching window of length m, the matching window includes a target character block, and is used to calculate the hash table, the jump table, the prefix table and the shift table, the index in each table represents a hash value calculated from the target character block, the shift in the shift table is the first jump value, the skip in the jump table is the second jump value, the id represents the number of a pattern string, the dark gray portion in the matching window represents the HB combination character block, and is used to initialize the bloom filter, which is essentially a bit array, and the bit values ​​include 0 and 1.

[0105] See also Figure 5 As shown in the pseudo code,

[0106] Algorithm 1 Scanning text matching algorithm

[0107]

[0108] After constructing the hash table, jump table, prefix table, shift table and Bloom filter, the four generated hash tables and Bloom filter are used to perform the scan matching stage. During the matching process, the matching window of length m is always kept moving to the right. Assuming that the character block with suffix length B of the current matching window is H, the initial index is set to m-1, the text length of the standard text is stored in the variable text_len, and the following steps are executed in a loop: If the index index is less than the text length text_len, the following steps are repeated: a: Extract the character block from the matching window to be checked: Get the target character block block_hash with a length of B on the right of the current window, that is, text[index-B:index]. Then execute b: use the shift table to judge and obtain the first jump value shift[H] corresponding to the target character block block_hash. If the first jump value shift[H] is greater than 0, directly skip the first jump value position and continue the next iteration process; if the first jump value shift[H] is not greater than 0, execute c: use Bloom filter to match, and perform Bloom filter verification on the target combined character block HB of length B composed of the suffix character block of the matching window and the prefix character block of the next matching window. If the verification passes, perform precise verification based on the prefix table prefix and the hash table; if the verification fails, obtain the second jump value jump[H] of the target character block in the jump table, jump to the second jump value jump[H] position, continue to search and verify from the standard text, until the standard text verification is completed, and obtain the text review result.

[0109] In the process of accurate verification based on the prefix table prefix and the hash table hash table, the prefix hash value prefix_hash of the current matching window can be first obtained, and the hash table and the prefix table can be used for accurate verification to find possible matches. If a match is found, the index and ID of the match are output, and the index is updated to index+jump[block_hash], and then the loop continues. When the index index reaches the text length of the standard text, it means that the standard text has been verified, and the loop stops, thereby obtaining the text review result.

[0110] In this embodiment, the optimal jump strategy can be pre-calculated in the preprocessing stage, thereby reducing the number of unnecessary character comparisons in the actual search process, thereby significantly reducing the time overhead in the actual matching process, especially when processing large volumes of text or a large number of keywords. This method effectively transfers some of the work that originally belonged to the matching stage to the preprocessing stage, thereby improving the overall performance.

[0111] Exemplarily, assuming that the pattern string set is P = {New Era, Once Upon a Time, Geometry Problems}, each pattern string in the pattern string set is subjected to CID encoding, and the encoded result is P = {079e85h679c1, 3f89463d8a09237f, 293b463d8a091c6373a2}. It can be seen that the shortest pattern string length value m = 12, the character block size value B = 8, and the hash table, jump table, prefix table, shift table and Bloom filter are obtained through preprocessing calculation. Among them, Figure 6 This is a schematic diagram of the structure of the constructed shift table, jump table and Bloom filter.

[0112] See also Figure 7 As shown, after constructing four hash tables and Bloom filter, the generated four hash tables and Bloom filter are used to perform the scan matching stage. First, the first jump value corresponding to the target character block 463d8a09 of the matching window in the displacement table is obtained. When the first jump value is 0, the precise verification will not be performed immediately. Instead, the suffix character block of the current matching window and the prefix character block of the next matching window are used to construct the target combination character block 8a09465c with a length of B. Then, the Bloom filter is used to verify the target combination character block 8a09465c in the standard text to obtain the verification result. When the verification fails, the standard text moves backward by the second jump value jump[463d8a09]=4 distances, and continues to check the standard text until the standard text verification result is obtained.

[0113] The standard text review method provided in the embodiment of the present application obtains a pattern string set, which includes multiple pattern strings; constructs a jump table and a hash table according to the suffix character blocks of all pattern string matching windows, and constructs a prefix table according to the prefix character blocks of the matching window, and constructs a displacement table according to the non-suffix character blocks of the matching window; obtains a combined character block, and constructs a Bloom filter according to the combined character block; the combined character block is composed of the suffix character block of the current matching window and the prefix character block of the next matching window; uses a hash table, a jump table, a prefix table, a displacement table and a Bloom filter to perform mobile scanning matching on the standard text through the matching window to be verified, and obtains a text review result. Compared with the prior art, this technical solution can dynamically perform matching jump processing according to the data in the displacement table by accurately constructing a displacement table according to the non-suffix character blocks of the matching window, thereby increasing the jump distance and increasing the jump amplitude, and by constructing a jump table according to the suffix character blocks of all pattern string matching windows, the jump value can be set in advance, and the time consumption of the matching stage can be transferred to the preprocessing stage, thereby speeding up the matching performance, and by accurately constructing a Bloom filter based on the combined character blocks, the character blocks can be preliminarily filtered to avoid the situation where precise matching is not required, thereby speeding up the matching process, and when the shortest pattern string is long and the same prefix and suffix exist in each pattern string, it has the advantages of a large jump distance and a small number of precise verification times, thereby further improving the efficiency of text review.

[0114] on the other hand, Figure 8 This is a schematic diagram of the structure of a standard text review device provided in an embodiment of the present application. The device may be a device in a terminal or a server, such as Figure 8 As shown, the device comprises:

[0115] The receiving module 710 is used to obtain a pattern string set; the pattern string set includes multiple pattern strings;

[0116] A first construction module 720 is used to construct a jump table and a hash table according to the suffix character blocks of all pattern string matching windows, to construct a prefix table according to the prefix character blocks of the matching windows, and to construct a displacement table according to the non-suffix character blocks of the matching windows;

[0117] The second construction module 730 is used to obtain a combined character block and construct a Bloom filter according to the combined character block; the combined character block is composed of a suffix character block of the current matching window and a prefix character block of the next matching window;

[0118] The review module 740 is used to use a hash table, a jump table, a prefix table, a displacement table and a Bloom filter to perform moving scan matching on the standard text through the matching window to be checked to obtain a text review result.

[0119] Optionally, the first building module 720 is specifically configured to:

[0120] Encode the pattern string to determine the number of pattern strings in the pattern string set and the length value of the shortest pattern string in the pattern string; the pattern string includes a suffix character block and a non-suffix character block;

[0121] Calculate the character block size value according to the number of pattern strings and the shortest pattern string length value;

[0122] Based on the character block size value, calculate the non-suffix character block and the first jump value corresponding to the non-suffix character block;

[0123] The non-suffix character block and the first jump value corresponding to the non-suffix character block are stored correspondingly to construct a displacement table, in which the first jump value corresponding to the suffix character block in the displacement table is zero.

[0124] Optionally, the second building module 730 is specifically configured to:

[0125] When the suffix character block is in the first situation, the second jump value is m-B+4; the first situation is that the suffix character block does not appear in the matching window of other pattern strings; the suffix character block is the character block currently being verified that is located at the rightmost end of the matching window and has a character block size of B; m is the shortest pattern string length value, and B is the character block size value;

[0126] When the suffix character block is in the second case, the second jump value is mB; the second case is that the suffix character block only appears at the leftmost end of other pattern strings;

[0127] When the suffix character block is in other situations, the second jump value is 4; other situations are situations other than the first and second situations;

[0128] A jump table is constructed based on the second jump value and the suffix character block.

[0129] Optionally, the review module 740 is specifically used to:

[0130] Obtain the target character block of the matching window to be verified and the first jump value of the target character block in the displacement table;

[0131] Determine whether the first jump value is greater than zero;

[0132] When the first jump value is not greater than zero, the suffix character block of the matching window and the prefix character block of the next matching window are used to construct a target combined character block;

[0133] A Bloom filter is used to verify the target combined character block in the standard text to obtain a verification result; the verification result is used to indicate a verification failure or a verification success;

[0134] According to the verification results, the prefix table, hash table and jump table are used to scan and match the standard text to obtain the text review results.

[0135] Optionally, the review module 740 is further configured to:

[0136] When the first jump value is greater than zero, the first jump value position is skipped;

[0137] Continue to search and verify from the standard text until the standard text is verified and the text review result is obtained.

[0138] Optionally, the review module 740 is further configured to:

[0139] When the verification result indicates that the verification is successful, the target character block in the matching window is verified based on the prefix table and the hash table to obtain the text review result;

[0140] When the verification result indicates that the verification has failed, the standard text is scanned and matched based on the jump table to obtain the text review result.

[0141] Optionally, the review module 740 is further configured to:

[0142] Get the second jump value of the target character block in the jump table;

[0143] Jump to the second jump value position, continue to search and verify from the standard text, until the standard text verification is completed, and obtain the text review result.

[0144] It can be understood that the functions of each functional module of the standard text review device of this embodiment can be specifically implemented according to the method in the above method embodiment, and its specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0145] On the other hand, see Fig. 9 As shown, the computer device provided in the embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the standard text review method as described above is implemented.

[0146] The computer device includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. Various programs and data required for system operation are also stored in the RAM 403. The CPU 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 406 is also connected to the bus 404.

[0147] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 406 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0148] In particular, according to the embodiments of the present disclosure, the above reference Figure 1 The described processes may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising a computer program for executing Figure 1 In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 409 and / or installed from the removable medium 411.

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units or modules involved in the embodiments described in the present application may be implemented by software or by hardware. The described units or modules may also be set in a processor. For example, it may be described as: a processor comprising: an acquisition module, a first construction module, a second construction module, and a review module. Among them, the names of these units or modules do not constitute a limitation on the units or modules themselves under certain circumstances. For example, the acquisition module may also be described as "used to obtain a pattern string set; the pattern string set includes multiple pattern strings".

[0151] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the standard text review method described in the present application:

[0152] Acquire a pattern string set; the pattern string set includes multiple pattern strings;

[0153] Constructing a jump table and a hash table according to the suffix character blocks of all the pattern string matching windows, constructing a prefix table according to the prefix character blocks of the matching windows, and constructing a displacement table according to the non-suffix character blocks of the matching windows;

[0154] Acquire a combined character block, and construct a Bloom filter according to the combined character block; the combined character block is composed of a suffix character block of a current matching window and a prefix character block of a next matching window;

[0155] The hash table, the jump table, the prefix table, the displacement table and the Bloom filter are used to perform mobile scanning matching on the standard text through the matching window to be verified to obtain a text review result.

[0156] In summary, the standard text review method, apparatus, device and medium provided in the embodiments of the present application obtain a pattern string set, which includes multiple pattern strings; build a jump table and a hash table based on the suffix character blocks of all pattern string matching windows, and build a prefix table based on the prefix character blocks of the matching window, and build a displacement table based on the non-suffix character blocks of the matching window; obtain a combined character block, and build a Bloom filter based on the combined character block; the combined character block is composed of the suffix character block of the current matching window and the prefix character block of the next matching window; use a hash table, a jump table, a prefix table, a displacement table and a Bloom filter to perform mobile scanning matching on the standard text through the matching window to be verified, and obtain a text review result. Compared with the prior art, this technical solution can dynamically perform matching jump processing according to the data in the displacement table by accurately constructing a displacement table according to the non-suffix character blocks of the matching window, thereby increasing the jump distance and increasing the jump amplitude, and by constructing a jump table according to the suffix character blocks of all pattern string matching windows, the jump value can be set in advance, and the time consumption of the matching stage can be transferred to the preprocessing stage, thereby speeding up the matching performance, and by accurately constructing a Bloom filter based on the combined character blocks, the character blocks can be preliminarily filtered to avoid the situation where precise matching is not required, thereby speeding up the matching process, and when the shortest pattern string is long and the same prefix and suffix exist in each pattern string, it has the advantages of a large jump distance and a small number of precise verification times, thereby further improving the efficiency of text review.

[0157] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0158] In addition, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc. Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware.

[0159] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. A standard text review method, characterized in that: The standard text review methods include: Acquire a pattern string set; the pattern string set includes multiple pattern strings; Constructing a jump table and a hash table according to the suffix character blocks of all the pattern string matching windows, constructing a prefix table according to the prefix character blocks of the matching windows, and constructing a displacement table according to the non-suffix character blocks of the matching windows; Acquire a combined character block, and construct a Bloom filter according to the combined character block; the combined character block is composed of a suffix character block of a current matching window and a prefix character block of a next matching window; The hash table, the jump table, the prefix table, the displacement table and the Bloom filter are used to perform mobile scanning matching on the standard text through the matching window to be verified to obtain a text review result.

2. The method according to claim 1, characterized in that Constructing a displacement table according to the non-suffix character blocks of the matching window, including: Encode the pattern string to determine the number of pattern strings in the pattern string set and the length value of the shortest pattern string in the pattern string; the pattern string includes a suffix character block and a non-suffix character block; Calculate the character block size value according to the number of pattern strings and the shortest pattern string length value; Based on the character block size value, calculating the non-suffix character block and a first jump value corresponding to the non-suffix character block; The non-suffix character block and the first jump value corresponding to the non-suffix character block are stored correspondingly to construct the displacement table, and the first jump value corresponding to the suffix character block in the displacement table is zero.

3. The method according to claim 2, characterized in that A jump table is constructed according to the suffix character blocks of all the pattern string matching windows, including: When the suffix character block is in the first situation, the second jump value is m-B+4; the first situation is that the suffix character block does not appear in the matching window of other pattern strings; the suffix character block is the character block currently being verified that is located at the rightmost end of the matching window and has a character block size of B; m is the shortest pattern string length value, and B is the character block size value; When the suffix character block is in the second situation, the second jump value is mB; the second situation is that the suffix character block only appears at the leftmost end of other pattern strings; When the suffix character block is in other situations, the second jump value is 4; the other situations are situations other than the first situation and the second situation; The jump table is constructed according to the second jump value and the suffix character block.

4. The method according to claim 1, characterized in that: The hash table, the jump table, the prefix table, the displacement table and the Bloom filter are used to perform mobile scanning matching on the standard text through the matching window to be checked to obtain a text review result, including: Obtain the target character block of the matching window to be verified and the first jump value of the target character block in the displacement table; Determining whether the first jump value is greater than zero; When the first jump value is not greater than zero, the suffix character block of the matching window and the prefix character block of the next matching window are used to construct a target combined character block; Using the Bloom filter to verify the target combined character block in the standard text to obtain a verification result; the verification result is used to indicate a verification failure or a verification success; According to the verification result, the prefix table, the hash table and the jump table are used to scan and match the standard text to obtain the text review result.

5. The method according to claim 4, characterized in that After determining whether the first jump value is greater than zero, the method further includes: When the first jump value is greater than zero, skipping the first jump value positions; Continue searching and verifying the standard text until the standard text is verified and the text review result is obtained.

6. The method according to claim 4, characterized in that According to the verification result, the prefix table, the hash table and the jump table are used to scan and match the standard text to obtain the text review result, including: When the verification result indicates that the verification is successful, the target character block in the matching window is verified based on the prefix table and the hash table to obtain the text review result; When the verification result indicates that the verification has failed, the standard text is scanned and matched based on the jump table to obtain the text review result.

7. The method according to claim 6, characterized in that Scanning and matching the standard text based on the jump table to obtain the text review result includes: Obtain a second jump value of the target character block in the jump table; Jump to the second jump value position, continue to search and verify from the standard text, until the standard text verification is completed, and obtain the text review result.

8. A standard text review device, characterized in that: The standard text review device comprises: An acquisition module, used to acquire a pattern string set; the pattern string set includes multiple pattern strings; A first construction module is used to construct a jump table and a hash table according to the suffix character blocks of all the pattern string matching windows, to construct a prefix table according to the prefix character blocks of the matching windows, and to construct a displacement table according to the non-suffix character blocks of the matching windows; A second construction module is used to obtain a combined character block and construct a Bloom filter according to the combined character block; the combined character block is composed of a suffix character block of a current matching window and a prefix character block of a next matching window; The review module is used to use the hash table, the jump table, the prefix table, the displacement table and the Bloom filter to perform mobile scanning matching on the standard text through the matching window to be verified to obtain a text review result.

9. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is used to implement the method according to any one of claims 1 to 7.

Images