Quick matching method for fuzzy keywords based on Trie tree
By combining a variant Trie tree with a global fail-fast bitmap and a node local reachability cache, the fuzzy matching process of the Trie tree is optimized, solving the performance bottleneck problem of traditional Trie trees in fuzzy matching and achieving efficient and accurate multi-dimensional similarity matching.
Patent Information
- Application Number
- CN202511820465.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-03-06
AI Technical Summary
Traditional Trie trees suffer from huge memory overhead and query performance bottlenecks due to the explosion of state space combinations during fuzzy matching, making them unable to effectively handle complex, multi-dimensional similarity matching needs.
A two-layer dynamic caching mechanism is adopted, which combines a variant Trie tree with a global fast failure bitmap and a local fast reachability cache for nodes. Combined with a multimodal similarity calculation module, the path exploration is optimized through fast failure determination and fast reachability query to avoid repeated similarity calculations.
It significantly improves query performance and memory efficiency, enhances matching accuracy and adaptability, and can effectively handle large-scale character sets and complex similarity definitions.
Smart Images
Figure CN121614652A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer information retrieval and data structure application technology, specifically, it relates to a fast matching method for fuzzy keywords based on Trie trees. Background Technology
[0002] In the field of modern information technology, string retrieval and matching are core foundational technologies supporting numerous application scenarios such as massive data processing, information security monitoring, and natural language understanding. With the exponential growth of network communication and data interaction, the rapid and accurate identification and filtering of keywords, especially those with various variations and errors, has become crucial for ensuring network content security, optimizing search engine experience, and improving human-computer interaction efficiency. Against this backdrop, various data structures for efficient indexing and querying of string sets have emerged. Among them, the Trie tree, also known as a dictionary tree or prefix tree, has gained widespread application and recognition in the field of exact matching due to its unique structural advantages.
[0003] Traditional Trie tree structures lack any fault tolerance for such deviations; a single character mismatch will halt the entire query process. To enable fuzzy matching in Trie trees, existing technologies typically employ a pre-expansion strategy. This involves attaching all possible similar variants of a character—homonyms, near-homophones, and misspelled letters—as parallel branches to the same parent node during tree construction. While this approach theoretically covers the needs of fuzzy matching, it introduces a more serious secondary problem in practice: combinatorial explosion of the state space. As the fault-tolerant edit distance increases or the set of similar characters expands, the tree width grows exponentially, leading to enormous memory overhead and construction time, making it impractical for handling large-scale character sets and multi-dimensional similarities. A deeper technical contradiction lies in the fact that this path expansion fundamentally alters the query model of Trie trees. The query process is no longer a linear traversal along a defined path, but degenerates into a complex, trial-and-error branch selection process at each level of the tree. For each character in the text to be matched, the algorithm needs to iteratively compare and calculate the similarity of each of the numerous child nodes of the current node in order to find the "most likely" path. This repetitive and costly similarity calculation at each node completely negates the original advantage of the Trie tree's O(L) time complexity. This causes the query performance to degenerate from being independent of the dictionary size to being strongly correlated with the branch width of the node, the computational complexity of the similarity function, and the complexity of the fault tolerance strategy. This forms a new and insurmountable performance bottleneck, which is particularly prominent in high-concurrency, low-latency real-time processing scenarios.
[0004] Therefore, how to organically integrate a matching mechanism that can handle complex, multi-dimensional similarity while retaining the advantages of prefix sharing and efficient index structure unique to Trie trees, and at the same time avoid repeating and expensive similarity calculations and invalid path explorations during each node traversal, thereby breaking the performance constraint between query efficiency and fault tolerance in traditional fuzzy matching methods, has become a key challenge and an urgent technical problem for those skilled in the art. Summary of the Invention
[0005] According to a first aspect of the present invention, the present invention claims protection for a fast matching method for fuzzy keywords based on a Trie tree, which is applied in a system architecture consisting of a data resource layer, a core index and matching layer, and a result adjudication layer, the method comprising the following steps: S1, Construct a variant Trie tree based on a preset keyword library, assign node identifiers to the nodes of the variant Trie tree, and initialize a global fast failure bitmap; S2, for the input target text to be matched, starting from each character position as the matching start position, the pointer of the current node of the variant Trie tree is reset to the root node of the variant Trie tree in sequence; S3, starting from the current matching start position, read the characters in the target text to be matched one by one as input characters. For each input character read and the current Trie tree node, execute fast failure judgment, fast reachability query and similarity calculation and path exploration in sequence. S41, if during the traversal of the similarity calculation and path exploration, at least one child node whose total similarity score with the input character exceeds a preset similarity threshold is obtained, then the child node with the highest score is selected as the best matching child node, the mapping relationship from the input character to the best matching child node is written into the node local fast reach cache of the current node, and the pointer of the current node is updated to the best matching child node.
[0006] Furthermore, S1 also includes: The variant Trie tree consists of a root node and several internal nodes and leaf nodes. Each non-root node is assigned a globally unique 32-bit unsigned integer node identifier within the variant Trie tree, and each node's data structure contains a local fast reachable cache. Initializes a global fail-fast bitmap that exists as a singleton in the system-wide scope to store 64-bit unsigned integers.
[0007] Furthermore, after performing fast failure determination, fast reachability query, and similarity calculation and path exploration in sequence in S3, it also includes: S42, if during the traversal of the similarity calculation and path exploration, no child node whose total similarity score with the input character exceeds the preset similarity threshold is found, then the 64-bit key value constructed in the fast failure determination is written into the global fast failure bitmap, and the matching attempt starting from the current matching start position is terminated.
[0008] Furthermore, S3, which sequentially executes fast failure determination, fast reachability query, and similarity calculation and path exploration, also includes: The fast failure determination is based on the 32-bit node identifier of the current node and the 32-bit numerical representation of the input character, constructing a unique 64-bit key value, and querying the global fast failure bitmap. If the query is successful, the current matching path is determined to be invalid, and the matching attempt starting from the current matching start position is immediately terminated. During a fast reachability query, if the fast failure determination fails, the local fast reachability cache of the current node is queried using the input character as the key. If the query is successful and returns a direct reference to a child node, the current node pointer is updated to the referenced child node, and the next input character is processed. During similarity calculation and path exploration, if neither the fast failure determination nor the fast reachability query yields a deterministic result, then all direct child nodes of the current node are traversed. For each child node, the multimodal similarity calculation module is called to calculate the total similarity score between the input character and the character associated with the child node.
[0009] Furthermore, in the fast failure determination step of S3, the construction method of the unique 64-bit key value is specifically as follows: The 32-bit unsigned integer node identifier of the current node is used as the high 32 bits, and the 32-bit numerical representation of the input character is used as the low 32 bits. They are then concatenated through bit operations to form a 64-bit unsigned long integer key. The global fast failure bitmap is a compressed and optimized bitmap data structure whose function is to globally record and share all combinations consisting of node identifiers and character values that have been identified as being unable to reach any valid path during the matching process.
[0010] Furthermore, each node of the variant Trie tree contains a local fast reachable cache, the data structure of which is a hash table that supports high-concurrency read and write. The key of the hash table is a 32-bit Unicode code point of a character in the input text, and the value is a direct pointer or reference to a specific child node of the current node. In the state update and node transfer steps, the operation of writing the mapping relationship into the local fast reach cache of the node dynamically stores and reuses the successfully calculated fuzzy matching path, so that when the same input characters are matched in the same current node in the future, the node transfer can be completed directly through the fast reach query, avoiding repeated similarity calculations.
[0011] Furthermore, the process of the multimodal similarity calculation module calculating the total similarity score specifically includes the step of calculating the sound-shape similarity score, comprising: Pinyin attribute acquisition: By querying a preset pinyin library, the pinyin triplet corresponding to each of the two Chinese characters to be compared is obtained. Each pinyin triplet consists of an initial consonant, a final vowel, and a tone value. The initial consonant similarity calculation is performed by querying a pre-constructed 23×23 initial consonant confusion matrix to obtain the similarity score between two initial consonants. The confusion matrix is based on linguistic statistics and presets a similarity score greater than 0 and less than 1.0 for initial consonant pairs that are easily confused in pronunciation. Vowel similarity calculation treats two vowels as strings, uses the Levenstein distance algorithm to calculate their edit distance, and then applies the formula: Similarity = 1 - (edit distance / maximum possible length of the two vowel strings); The edit distance is normalized to obtain the vowel similarity score; Tone similarity is calculated by comparing the tone values of two pinyin syllables. If the tone values are the same, the tone similarity is 1.0; otherwise, it is 0.0. The phonetic similarity score is obtained by weighted summation of the calculated initial similarity, final similarity, and tone similarity using a first preset weighting coefficient.
[0012] Furthermore, the process of calculating the total similarity score by the multimodal similarity calculation module also includes calculating the character shape similarity score, and comprehensively weighting the phonetic similarity score and the character shape similarity score, wherein calculating the character shape similarity score includes: The character shape encoding is obtained by querying the Wubi encoding library and the Zhengma encoding library respectively to obtain the Wubi character root sequence and the Zhengma character root sequence corresponding to the two Chinese characters to be compared. The initial character shape similarity calculation involves converting the Wubi character root sequence and the Zheng code character root sequence into word frequency vectors respectively, and using the cosine similarity algorithm to calculate the cosine value of the angle between the Wubi encoded word frequency vectors and the Zheng code encoded word frequency vectors of the two characters respectively, to obtain the initial Wubi character shape similarity and the initial Zheng code character shape similarity. The stroke difference penalty calculation involves retrieving the stroke counts of two characters from a Chinese character stroke database, calculating the absolute value Δ of the difference in their stroke counts, and then applying a truncated linear penalty function. Penalty value = min(Δ,T) / D; To calculate the penalty term, where T is the preset cutoff threshold and D is the preset normalized denominator; The character shape similarity is corrected by subtracting the calculated stroke difference penalty value from the initial Wubi character shape similarity and the initial Zhengma character shape similarity, respectively, to obtain the corrected Wubi character shape similarity and the corrected Zhengma character shape similarity. The total similarity score is obtained by weighting the obtained phonetic and shape similarity scores, as well as the obtained corrected Wubi character shape similarity and corrected Zhengma character shape similarity scores, and then summing them for the second time using a second preset weighting coefficient.
[0013] Compared with the prior art, the present invention has the following beneficial effects: Significantly improved query performance: By introducing a two-layer dynamic caching mechanism consisting of a local fast reachable cache for nodes and a global fast failure bitmap, this invention transforms the computationally expensive similarity comparison process, which must be performed at each node, into a decision-making process that, in most cases, only requires two efficient hash table or bitmap queries. Successful paths are dynamically memorized, and failed paths are globally excluded, greatly reducing redundant computation and reducing the average time complexity of fuzzy matching by orders of magnitude. The performance advantage is particularly prominent when dealing with frequently occurring character combinations and large-scale text.
[0014] High memory efficiency and system scalability: This invention abandons the traditional approach of pre-expanding the Trie tree in fuzzy matching schemes. The core Trie tree structure is constructed solely based on the original keyword set, maintaining its inherent compactness. Fuzzy matching capabilities are achieved through dynamic computation and caching, rather than static structural expansion, thus fundamentally solving the problem of state space combinatorial explosion. This allows the method to effectively support large-scale character sets and complex similarity definitions while maintaining low memory consumption.
[0015] High matching accuracy and adaptability: The multimodal similarity calculation module designed in this invention comprehensively considers the phonetic and visual features of Chinese characters, and innovatively introduces a consonant confusion matrix and a stroke difference penalty mechanism. This enables more accurate simulation of human perception of character similarity, effectively identifying various variant words, similar-looking words, and similar-sounding words, thus improving matching accuracy and recall. Furthermore, the multi-level weight coefficients in the algorithm are all configurable parameters, allowing this method to flexibly adapt to different business scenarios with varying emphases on phonetic and visual similarity.
[0016] The architecture is clear and easy to implement: The technical solution proposed in this invention clearly modularizes and decouples the core data structure, acceleration mechanism, similarity algorithm, and business adjudication logic, forming a four-in-one system architecture of multi-database collaboration, variant Trie tree, global failure bitmap, and similarity algorithm. This architecture is logically rigorous and clearly defined in its responsibilities, which not only facilitates engineering implementation and maintenance but also provides a solid foundation for future functional expansion or performance optimization. Attached Figure Description
[0017] Figure 1 A flowchart illustrating the workflow of a fast matching method for fuzzy keywords based on a Trie tree, as claimed in an embodiment of the present invention. Figure 2 The flowchart shows the query operation of a global fast failure bitmap for a fast matching method for fuzzy keywords based on a Trie tree, as claimed in an embodiment of the present invention. Figure 3 A schematic diagram illustrating the rapid node reachability of a fast matching method for fuzzy keywords based on a Trie tree, as claimed in an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the creation process of a Trie tree for a fast matching method of fuzzy keywords based on a Trie tree, as claimed in an embodiment of the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0020] As a multi-branch tree structure, the core design idea of a Trie tree is to reduce storage overhead and query redundancy by utilizing the common prefixes of string sequences. In this structure, the path from the root node to any node constitutes a string prefix, and each edge of the tree corresponds to a specific character. This design allows Trie trees to significantly compress storage space when processing large-scale dictionaries by sharing prefixes, and its query time complexity is only related to the length of the query string, not the size of the dictionary, thus providing excellent performance guarantees for accurate keyword matching and prefix lookup tasks.
[0021] However, with the continuous development of related technologies and the increasingly stringent performance requirements of application scenarios, especially in the fields of fuzzy matching and error-tolerant recognition, the inherent characteristics of the aforementioned Trie tree technology based on precise character mapping have gradually revealed its inherent limitations in addressing new challenges. The construction and querying process of a Trie tree is essentially a deterministic finite state automaton, where the state transition path is rigid, meaning that each input character strictly corresponds to a definite path branch. This one-to-one mapping relationship is the cornerstone of its high-speed and accurate matching, but it also constitutes a fundamental obstacle preventing it from directly adapting to fuzzy matching needs. When dealing with scenarios such as user input errors, homophones, and variant words deliberately circumventing censorship, there are often discrepancies between the characters in the query string and the characters on the Trie tree node path.
[0022] According to the first embodiment of the present invention, referring to Figure 1 This invention claims protection for a fast matching method for fuzzy keywords based on a Trie tree, which is applied to a system architecture consisting of a data resource layer, a core index and matching layer, and a result adjudication layer. The method includes the following steps: S1, Construct a variant Trie tree based on a preset keyword library, assign node identifiers to the nodes of the variant Trie tree, and initialize a global fast failure bitmap; S2, for the input target text to be matched, starting from each character position as the matching start position, the pointer of the current node of the variant Trie tree is reset to the root node of the variant Trie tree in sequence; S3, starting from the current matching start position, read the characters in the target text to be matched one by one as input characters. For each input character read and the current Trie tree node, execute fast failure judgment, fast reachability query and similarity calculation and path exploration in sequence. S41. If, during the traversal of the similarity calculation and path exploration, at least one child node is obtained whose total similarity score of its associated characters with the input character exceeds the preset similarity threshold, then select the child node with the highest score as the best matching child node, write the mapping relationship from the input character to the best matching child node into the node local fast reachable cache of the current node, and update the current node pointer to the best matching child node.
[0023] Among them, in this embodiment, the data resource layer is the data cornerstone of the entire matching method, and its physical form is a set of data file collections stored in a non-volatile storage medium and loaded into high-speed memory during the system initialization phase.
[0024] Specifically, the data resource layer includes five core databases: a keyword database, a pinyin database, a Wubi coding database, a Zhengma coding database, and a Chinese character stroke database. The keyword database is stored in UTF-8 encoding format, and each line contains a target keyword string to be indexed, such as "Peking University", etc. It is the direct input source for constructing the core index structure. The pinyin database is responsible for establishing an accurate mapping between Chinese characters and Chinese pinyin. Its data structure is a set of key-value pairs, where the key is the Chinese character Unicode code point represented by a 32-bit unsigned integer, and the value is a list containing one or more pinyin triples to handle the case of polyphonic characters. Each pinyin triple consists of an initial consonant string, a final consonant string, and an integer tone value. For example, for the Chinese character "行", its corresponding entry in the pinyin database may be [('h', 'ang', 2), ('x', 'ing', 2)]. The structures of the Wubi coding database and the Zhengma coding database are similar to that of the pinyin database, also using the Chinese character Unicode code point as the key, and the value is a list of strings of its corresponding Wubi root coding sequence or Zhengma root coding sequence. The Chinese character stroke database is a more concise mapping table that directly maps the Chinese character Unicode code point to an 8-bit unsigned integer representing its total number of strokes. These five databases together provide complete and standardized basic data support for subsequent multi-modal similarity calculation and index construction.
[0025] Furthermore, the core index and matching layer 20, which is the main body of the technical solution of the present invention, is explained. This layer is responsible for constructing indexes, performing matching, and applying acceleration strategies, and is internally composed of three highly coupled modules: a variant Trie tree module, a global fast failure bitmap module, and a multi-modal similarity calculation module.
[0026] The core of the variant Trie tree module is an index structure that enhances the functionality of the standard Trie tree data structure. This variant Trie tree consists of a root node and a series of internal nodes and leaf nodes linked by parent-child relationships. Except for the root node, each node is explicitly associated with a Chinese character. In a concrete engineering implementation, the data structure of a node object contains the following member variables: a pointer to the set of child nodes, which can be implemented as a hash table with the 32-bit Unicode code points of the characters associated with the child nodes as keys and pointers to child node objects as values.
[0027] The result adjudication layer includes an adaptive threshold adjudication mechanism module responsible for the final screening and confirmation of the candidate keyword set output by the core index and the matching layer. For each matched candidate keyword, this module comprehensively considers its actual similarity score for each character during the matching process, the keyword's length, and statistical features such as keyword importance or frequency of use possibly imported from external knowledge bases, calculating a final confidence score using a predefined scoring function. This score is then compared to a dynamically set adjudication threshold. This threshold can be dynamically adjusted according to the system's different requirements for precision and recall. Only keywords with a confidence score higher than the current threshold are output as the final matching result.
[0028] Furthermore, S1 also includes: The variant Trie tree consists of a root node and several internal nodes and leaf nodes. Each non-root node is assigned a globally unique 32-bit unsigned integer node identifier within the variant Trie tree, and each node's data structure contains a local fast reachable cache. Initializes a global fail-fast bitmap that exists as a singleton in the system-wide scope to store 64-bit unsigned integers.
[0029] Furthermore, after performing fast failure determination, fast reachability query, and similarity calculation and path exploration in sequence in S3, it also includes: S42, if during the traversal of the similarity calculation and path exploration, no child node whose total similarity score with the input character exceeds the preset similarity threshold is found, then the 64-bit key value constructed in the fast failure determination is written into the global fast failure bitmap, and the matching attempt starting from the current matching start position is terminated.
[0030] Furthermore, S3, which sequentially executes fast failure determination, fast reachability query, and similarity calculation and path exploration, also includes: The fast failure determination is based on the 32-bit node identifier of the current node and the 32-bit numerical representation of the input character, constructing a unique 64-bit key value, and querying the global fast failure bitmap. If the query is successful, the current matching path is determined to be invalid, and the matching attempt starting from the current matching start position is immediately terminated. During a fast reachability query, if the fast failure determination fails, the local fast reachability cache of the current node is queried using the input character as the key. If the query is successful and returns a direct reference to a child node, the current node pointer is updated to the referenced child node, and the next input character is processed. During similarity calculation and path exploration, if neither the fast failure determination nor the fast reachability query yields a deterministic result, then all direct child nodes of the current node are traversed. For each child node, the multimodal similarity calculation module is called to calculate the total similarity score between the input character and the character associated with the child node.
[0031] In this embodiment, under UTF-8 encoding, a Chinese character occupies a maximum of 4 bytes; a 32-bit unsigned integer also occupies 4 bytes, and the two combined total 8 bytes, which can be used as a 64-bit long integer. Treating the 32 bits as an unsigned number allows for the representation of 4,294,967,295 nodes, sufficient for each node in the Trie to correspond to any Chinese character. To simultaneously ensure the lookup speed and memory efficiency of the 64-bit bitmap, a high-performance compressed bitmap such as Roaring64Bitmap can be directly used.
[0032] Reference Figure 2 The query process for a global fast-fail bitmap includes: Convert the original string and the Trie tree node ID into 4-byte arrays respectively; Merge these two arrays into an 8-byte array, using the Trie tree node ID byte array as the high 4 bits and the original character byte array as the low 4 bits; Convert the merged array to a long integer and look up the global bitmap.
[0033] Saving the global fail-fast bitmap: Convert the original string and the unreachable Trie tree node ID into 4-byte arrays respectively; Merge these two arrays into an 8-byte array, using the Trie tree node ID byte array as the high 4 bits and the original character byte array as the low 4 bits; The merged array is converted into a long integer and written to the global bitmap.
[0034] For example: the original character is "wolf", the Trie tree node id is 5, after being converted into byte arrays, they are [-25, -117, -68] and [0, 0, 0, 5], after being merged, [0, 0, 0, 5, 0, -25, -117, -68], which is converted into a long integer 21490011068.
[0035] Furthermore, in the fast failure determination step of S3, the construction method of the unique 64-bit key value is specifically as follows: The 32-bit unsigned integer node identifier of the current node is used as the high 32 bits, and the 32-bit numerical representation of the input character is used as the low 32 bits. They are then concatenated through bit operations to form a 64-bit unsigned long integer key. The global fast failure bitmap is a compressed and optimized bitmap data structure whose function is to globally record and share all combinations consisting of node identifiers and character values that have been identified as being unable to reach any valid path during the matching process.
[0036] In this embodiment, the global fail-fast bitmap is another core acceleration mechanism of the present invention. It acts as a global negative cache in the system architecture and is implemented as a global object using a singleton design pattern. Internally, it encapsulates a highly compressed and optimized bitmap data structure specifically for storing 64-bit unsigned integers. The function of this bitmap is to record all "state-character" combinations that are determined to be unable to reach any valid path during the matching process. Specifically, a "state" is uniquely determined by the node_id of a node in the variant Trie tree. When the matching process reaches a node N, if the input text character C fails to find any match that meets the preset similarity threshold after comparing its similarity with the characters associated with all child nodes of N, the system determines that character C is a "dead end" in the state of node N. At this time, the system combines the 32-bit node_id of node N with the 32-bit Unicode code point of character C to form a unique 64-bit unsigned long integer key. This combination process is precisely implemented using bitwise operations: `(uint64_t)node_id << 32 | (uint64_t)C_unicode`, which places the node identifier in the high 32 bits and the character code point in the low 32 bits. This generated 64-bit key is then placed into the global fail-fast bitmap. In any subsequent matching task, regardless of the thread or matching context, when the system needs to process the character C at node N, it first generates the 64-bit key in the same way and queries the global bitmap. If the query result is a match, it means that the path has previously been verified as invalid, and the system can immediately prune the path, terminating the current matching path exploration, thus avoiding repeated and invalid computational investment on known failed paths.
[0037] Furthermore, each node of the variant Trie tree contains a local fast reachable cache, the data structure of which is a hash table that supports high-concurrency read and write. The key of the hash table is a 32-bit Unicode code point of a character in the input text, and the value is a direct pointer or reference to a specific child node of the current node. In the state update and node transfer steps, the operation of writing the mapping relationship into the local fast reach cache of the node dynamically stores and reuses the successfully calculated fuzzy matching path, so that when the same input characters are matched in the same current node in the future, the node transfer can be completed directly through the fast reach query, avoiding repeated similarity calculations.
[0038] In this embodiment, reference is made to Figure 3 The variant Trie tree is constructed once using a keyword library. Each node has a built-in concurrent hash table <character, child nodes>, which is the Quick-Reach Cache (QRC) that supports multi-threaded concurrent access.
[0039] The purpose of QRC (Quick Reference Count) is to: when the original character and a child node character are determined to be similar using the "phonetic-character-shape similarity matching algorithm," the system immediately writes the shortcut "original character → child node" into the parent node's QRC. The next time the same original character is encountered, a direct lookup in the table will jump to the child node, eliminating the need to recalculate the similarity.
[0040] The following demonstrates this process: First time: The original character successfully matches the character carried by node 2 through the "sound-character similarity matching algorithm", so the record "original character → node 2" is written into the QRC of node 1; For the second and subsequent times: the same character can be directly retrieved from the QRC of node 1 to node 2 in one step, completing the process instantly.
[0041] Furthermore, the process of the multimodal similarity calculation module calculating the total similarity score specifically includes the step of calculating the sound-shape similarity score, comprising: Pinyin attribute acquisition: By querying a preset pinyin library, the pinyin triplet corresponding to each of the two Chinese characters to be compared is obtained. Each pinyin triplet consists of an initial consonant, a final vowel, and a tone value. The initial consonant similarity calculation is performed by querying a pre-constructed 23×23 initial consonant confusion matrix to obtain the similarity score between two initial consonants. The confusion matrix is based on linguistic statistics and presets a similarity score greater than 0 and less than 1.0 for initial consonant pairs that are easily confused in pronunciation. Vowel similarity calculation treats two vowels as strings, uses the Levenstein distance algorithm to calculate their edit distance, and then applies the formula: Similarity = 1 - (edit distance / maximum possible length of the two vowel strings); The edit distance is normalized to obtain the vowel similarity score; Tone similarity is calculated by comparing the tone values of two pinyin syllables. If the tone values are the same, the tone similarity is 1.0; otherwise, it is 0.0. The phonetic similarity score is obtained by weighted summation of the calculated initial similarity, final similarity, and tone similarity using a first preset weighting coefficient.
[0042] Reference Figure 4 The process of creating a Trie tree includes: Trie tree initialization: Create an empty root node.
[0043] Traverse the keyword lexicon. For each word to be inserted in the lexicon, process it character by character, starting from the first character.
[0044] Create a node: Starting from the root node, check if the current character exists in the nexts dictionary of the current node.
[0045] If it exists, continue down the path of that character and update the current node to the child node corresponding to that character.
[0046] If it does not exist: Create a new Node.
[0047] Make this Node hold the character.
[0048] Assign a globally unique 32-bit integer id to the Node.
[0049] Initialize a thread-safe, concurrently available empty hash table inside the Node for use as a Quick-Reach Cache (QRC).
[0050] Will<character, Node> The key-value pair is inserted into the nexts dictionary of the current node.
[0051] Update the current node to this newly created child node.
[0052] Next character: Repeat step 3 to process the next character of the word until all characters of the word have been processed.
[0053] Mark the end: After all characters of a word have been inserted, the current node represents the last character of that word. At this point, the `end` flag of that node needs to be set to `True`.
[0054] Furthermore, the process of calculating the total similarity score by the multimodal similarity calculation module also includes calculating the character shape similarity score, and comprehensively weighting the phonetic similarity score and the character shape similarity score, wherein calculating the character shape similarity score includes: The character shape encoding is obtained by querying the Wubi encoding library and the Zhengma encoding library respectively to obtain the Wubi character root sequence and the Zhengma character root sequence corresponding to the two Chinese characters to be compared. The initial character shape similarity calculation involves converting the Wubi character root sequence and the Zheng code character root sequence into word frequency vectors respectively, and using the cosine similarity algorithm to calculate the cosine value of the angle between the Wubi encoded word frequency vectors and the Zheng code encoded word frequency vectors of the two characters respectively, to obtain the initial Wubi character shape similarity and the initial Zheng code character shape similarity. The stroke difference penalty calculation involves retrieving the stroke counts of two characters from a Chinese character stroke database, calculating the absolute value Δ of the difference in their stroke counts, and then applying a truncated linear penalty function. Penalty value = min(Δ,T) / D; To calculate the penalty term, where T is the preset cutoff threshold and D is the preset normalized denominator; The character shape similarity is corrected by subtracting the calculated stroke difference penalty value from the initial Wubi character shape similarity and the initial Zhengma character shape similarity, respectively, to obtain the corrected Wubi character shape similarity and the corrected Zhengma character shape similarity. The total similarity score is obtained by weighting the obtained phonetic and shape similarity scores, as well as the obtained corrected Wubi character shape similarity and corrected Zhengma character shape similarity scores, and then summing them for the second time using a second preset weighting coefficient.
[0055] In this embodiment, the multimodal similarity calculation module is crucial for ensuring the accuracy and robustness of fuzzy matching in this invention. This module encapsulates a composite similarity calculation algorithm that takes two Chinese characters as input and outputs a normalized floating-point number between 0.0 and 1.0 to quantify the degree of similarity between them. This algorithm comprehensively measures the similarity from two macroscopic dimensions: the phonetic and graphical representation of the Chinese characters.
[0056] The calculation process for phonetic similarity is detailed as a step-by-step comparison of each component of the pinyin. First, the module queries the pinyin database in data resource layer 10 to obtain the initial consonant, final vowel, and tone for each of the two input characters C1 and C2. Considering the existence of polyphonic characters, this step may return multiple triplets; the one that results in the highest final similarity is selected for subsequent processing. Second, initial consonant similarity is calculated. This process is not a simple equality check, but rather completed by querying a pre-constructed 23×23 initial consonant confusion matrix. This matrix is constructed based on statistical analysis of a large-scale corpus and linguistic knowledge; its diagonal elements all have a value of 1.0, indicating the highest similarity among themselves. Non-diagonal pairs of specific elements, such as ('n', 'l'), ('zh', 'z'), and ('f', 'h')—consonant pairs that are easily confused in pronunciation—are assigned positive values less than 1.0 but greater than 0. For example, sim('n', 'l') = 0.9, and sim('zh', 'z') = 0.85. Next, vowel similarity is calculated. This process treats vowels (such as 'ang', 'iang', 'ue') as strings and uses the standard Levenstein distance algorithm to calculate the edit distance d between two vowel strings. The resulting distance is normalized using the formula Similarity_final = 1.0 - (d / max_len), where max_len is the maximum length of the two vowel strings being compared. Subsequently, tone similarity is calculated, which is a binary judgment: if the tone values of the two pinyin are exactly the same, the tone similarity is 1.0; otherwise, it is 0.0. Finally, the calculated initial consonant similarity S_initial, final vowel similarity S_final, and tone similarity S_tone are weighted and summed using a set of preset, configurable weighting coefficients to obtain the final phonetic similarity score. In a preferred implementation configuration, the weights w_i, w_f, and w_t are set to 0.4, 0.4, and 0.2, respectively.
[0057] The calculation process of glyph similarity is more complex, which combines two features: Chinese character encoding structure and stroke contour. First, the module separately queries the Wubi encoding library and Zhengma encoding library in the data resource layer 10 to obtain the Wubi root sequences and Zhengma root sequences of two input characters C1 and C2. Secondly, for the root sequences of each encoding, the system converts them into a high-dimensional term frequency vector. The dimension of this vector is equal to the total number of all basic roots in this encoding scheme. Each element in the vector corresponds to a basic root, and its value is the number of times this root appears in the character encoding sequence. Then, the system uses the cosine similarity algorithm to calculate the cosine value of the angle between the Wubi encoding term frequency vectors of C1 and C2 respectively, obtaining the initial Wubi glyph similarity S_wubi; similarly, calculate the cosine similarity of the Zhengma encoding term frequency vectors of the two to obtain the initial Zhengma glyph similarity S_zhengma. To correct the misjudgment that may be caused by some characters having similar encodings but very different actual glyph contours (such as "疽" and "龘"), the system introduces a stroke difference penalty mechanism. This mechanism queries the Chinese character stroke library to obtain the stroke numbers N1 and N2 of the two characters, calculates the absolute value of their difference Δ = |N1 - N2|, and applies a truncated linear penalty function Penalty = min(Δ, T) / D to calculate the penalty term. In a specific embodiment, the truncation threshold T is set to 10, and the normalization denominator D is set to 30. This means that when the stroke difference is within 10 strokes, the penalty intensity increases linearly with the difference, and when it exceeds 10 strokes, the penalty intensity remains at the maximum value of 1 / 3. Finally, subtract this penalty value from the initial Wubi and Zhengma glyph similarities respectively to obtain the corrected two glyph similarities: S'_wubi = S_wubi - Penalty and S'_zhengma = S_zhengma - Penalty.
[0058] Finally, the multi-modal similarity calculation module performs a second weighted summation on the phonetic and glyph similarities Score_phonetic, the corrected Wubi glyph similarity S'_wubi, and the corrected Zhengma glyph similarity S'_zhengma calculated above through another set of weight coefficients that can be configured according to specific application scenarios, so as to obtain a comprehensive and integrated total similarity score . In a set of preferred implementation configurations, the weights w_p, w_wb, and w_zm are respectively set to 0.5, 0.25, and 0.25. This Score_total is the final output of the module and serves as the direct basis for subsequent matching decisions.
[0059] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
[0060] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
[0061] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.
Claims
1. A fast fuzzy keyword matching method based on Trie tree, which is applied to a system architecture composed of a data resource layer, a core index and matching layer, and a result decision layer, characterized in that, The method comprises the following steps: S1, based on a preset keyword library, a variant Trie tree is constructed, a node identifier is assigned to a node of the variant Trie tree, and a global fast failure bitmap is initialized; S2, for an input target text to be matched, a pointer of a current node of the variant Trie tree is reset to a root node of the variant Trie tree from each character position as a matching starting position; S3, starting from the current matching starting position, characters in the target text to be matched are read one by one as input characters, and for each read input character and the current Trie tree node, fast failure judgment, fast reachability query and similarity calculation and path exploration are sequentially performed; S41, if at least one child node whose associated character and the input character have a total similarity score exceeding a preset similarity threshold is obtained in the traversal process of the similarity calculation and path exploration, a child node with the highest score is selected as a best matching child node, a mapping relationship from the input character to the best matching child node is written into a node local fast reachability cache of the current node, and the current node pointer is updated to the best matching child node.
2. The method of claim 1, wherein, The S1 further comprises: The variant Trie tree is composed of a root node and a plurality of internal nodes and leaf nodes, wherein each non-root node is assigned a 32-bit unsigned integer node identifier which is globally unique in the variant Trie tree, and the data structure of each node contains a node local fast reachability cache; A global fast failure bitmap for storing 64-bit unsigned integers is initialized to exist as a singleton in the system global scope.
3. The method of claim 2, wherein, After the fast failure judgment, the fast reachability query and the similarity calculation and path exploration are sequentially performed in the S3, the following further comprises: S42, if no child node whose associated character and the input character have a total similarity score exceeding the preset similarity threshold is found in the traversal process of the similarity calculation and path exploration, a 64-bit key value constructed in the fast failure judgment is written into the global fast failure bitmap, and the matching attempt starting from the current matching starting position is terminated.
4. The method of claim 2, wherein, The fast failure judgment, the fast reachability query and the similarity calculation and path exploration are sequentially performed in the S3, and the following further comprises: The fast failure judgment, the fast reachability query and the similarity calculation and path exploration are sequentially performed in the S3, and the following further comprises: The fast failure judgment, the fast reachability query and the similarity calculation and path exploration are sequentially performed in the S3, and the following further comprises: The fast failure judgment, the fast reachability query and the similarity calculation and path exploration are sequentially performed in the S3, and the following further comprises: The fast failure judgment, the fast reachability query and the similarity calculation and path exploration are sequentially performed in the S3, and the following further comprises: In the similarity calculation and path exploration, if the fast failure determination and the fast reachability query do not obtain a deterministic result, all direct child nodes of the current node are traversed, for each child node, a multi-modal similarity calculation module is called to calculate the total similarity score between the input character and the character associated with the child node.
5. The method of claim 2, wherein, In the fast failure determination step of the S3, the unique 64-bit key value is constructed in the following manner: The 32-bit unsigned integer node identifier of the current node is taken as the high 32 bits, and the 32-bit numerical representation of the input character is taken as the low 32 bits, which are spliced through bit operations to form a 64-bit unsigned long integer key; The global fast failure bitmap is a compressed and optimized bitmap data structure, which globally records and shares all combinations of node identifiers and character numerical representations that have been confirmed as unable to lead to any valid path in the matching process.
6. The method of claim 2, wherein, The node-local fast reachability cache contained in each node of the variant Trie tree is a hash table that supports high-concurrency read and write, the key of the hash table is the 32-bit Unicode code point of the character in the input text, and the value is a direct pointer or reference to a certain determined child node of the current node; In the state updating and node transfer step, the mapping relationship is written into the node-local fast reachability cache, which dynamically stores and reuses the successfully calculated fuzzy matching paths, so that when the same input character is encountered in the matching at the same current node, the node transfer is directly completed through the fast reachability query, avoiding repeated similarity calculation.
7. The method of claim 4, wherein, The process of calculating the total similarity score by the multi-modal similarity calculation module specifically includes the step of calculating the phonetic-visual similarity score, which includes: Pinyin attribute acquisition: by querying a pre-set pinyin library, the pinyin triplets corresponding to each of the two Chinese characters to be compared are obtained, each pinyin triplet is composed of an initial, a final, and a tone value; Initial similarity calculation: by querying a pre-constructed 23x23 initial confusion matrix, the similarity score between the two initials is obtained, wherein the confusion matrix is based on linguistic statistical data and pre-sets a similarity score greater than 0 and less than 1.0 for initial pairs that are easily confused in pronunciation; Final similarity calculation: two finals are treated as strings, the edit distance is calculated using the Levenshtein distance algorithm, and the final similarity score is obtained by normalizing the edit distance through the formula: Similarity = 1 - (edit distance / maximum possible length of two final strings); Tone similarity calculation: compare the tone values of the two pinyins, if the tone values are the same, the tone similarity is 1.0, otherwise it is 0.0; Phonetic-visual similarity weighted summation: the calculated initial similarity, final similarity, and tone similarity are weighted and summed through a first pre-set weight coefficient to obtain the final phonetic-visual similarity score. 8. The method of claim 7, wherein, The multi-modal similarity calculation module calculates the total similarity score, and the process further includes calculating a graph similarity score, and integrating and weighting the phonetic-graph similarity score and the graph similarity score, wherein the calculation of the graph similarity score includes: obtaining graph codes, respectively querying a Wubi code library and a Zheng code library to obtain Wubi root sequences and Zheng root sequences corresponding to the two characters to be compared; initial graph similarity calculation, converting the Wubi root sequences and the Zheng root sequences into word frequency vectors, respectively, and using a cosine similarity algorithm to calculate the cosine values of the angles between the Wubi code word frequency vectors and the Zheng code word frequency vectors, respectively, to obtain initial Wubi graph similarity and initial Zheng graph similarity; stroke difference penalty calculation, querying a stroke library to obtain the stroke counts of the two characters, calculating the absolute value Δ of the stroke count difference, and applying a truncated linear penalty function: penalty value = min(Δ, T) / D; to calculate the penalty term, where T is a preset truncation threshold and D is a preset normalization denominator; graph similarity correction, subtracting the calculated stroke difference penalty value from the obtained initial Wubi graph similarity and initial Zheng graph similarity, respectively, to obtain corrected Wubi graph similarity and corrected Zheng graph similarity; total similarity integrated weighting, obtaining the phonetic-graph similarity score, the corrected Wubi graph similarity, and the corrected Zheng graph similarity, and performing a second weighting summation through a second preset weight coefficient to obtain the total similarity score.