A method and apparatus for optimizing fuzzy matching in a large-scale KV database monitoring system.

Abstract

This invention provides a method and apparatus for optimizing fuzzy matching in a large-scale KV database monitoring process. The method includes: defining a separator for the monitored keys, and segmenting each key's module, type, instance, and keyword using this separator; when a client monitors a key, finding the last separator, calculating the hash value of the string, and saving the linked lists corresponding to the module, type, and instance to a hash table, where the hash table's value is a linked list used to store keywords; when the database updates a key, first finding the last separator in the key, and using the separator to segment the key into keyPre1 and keyPost; checking if a linked list of keyPre1 exists in the hash table; if no linked list of keyPre1 exists, checking if keyPre1 contains a separator; if no separator exists, the process ends; if a separator exists, using the separator to segment keyPre1, continuing until no separator exists in the key. This method reduces key matching time and CPU usage, improving the performance of the KV database.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of communication technology, and in particular to a fuzzy matching optimization method and apparatus for large-scale monitoring of KV databases. Background Technology

[0002] A key-value database is a database that uses key-value storage, where data is organized, indexed, and stored in key-value pairs. Key-value storage is well-suited for data that doesn't involve numerous data relationships or business logic, effectively reducing disk read / write operations. It offers better read / write performance than SQL databases and solves the problem of relational databases' inability to store data structures.

[0003] In the most commonly used key-value database, Redis, the subscription relationship between clients and channels is stored using a dictionary plus linked list structure. In this case, if the number of subscribing clients increases, it means that every update, deletion, or creation of a data key requires scanning and matching this linked list to check if the key is being monitored, resulting in significant performance overhead. Therefore, for scenarios requiring a large number of monitoring functions, using Redis causes a significant amount of CPU consumption to be spent on key matching. Thus, a method is needed to greatly reduce key matching time and CPU usage under heavy monitoring conditions. Summary of the Invention

[0004] To address the above issues, this invention splits the key and stores the listening client in a hash + linked list. This significantly reduces key matching time and CPU usage, thereby improving the performance of the KV database under heavy listening conditions.

[0005] According to an embodiment of the present invention, a method and apparatus for optimizing fuzzy matching for large-scale monitoring of a KV database are provided.

[0006] In a first aspect of the invention, a method for optimizing fuzzy matching in a large-scale KV database monitoring system is provided. The method includes:

[0007] S01: Specifies the delimiter for the listening key, and uses this delimiter to segment the module, type, instance, and keyword of each key;

[0008] S02: When the client listens for the key, it finds the last delimiter, calculates the hash value of the string, and saves the linked lists corresponding to the module, type, and instance to the hash table. The value of the hash table is a linked list used to store the key.

[0009] S03: When updating a key in the database, first find the last delimiter in the key, and use the delimiter to split the key into: keyPre1 and keyPost. The keyPre1 refers to the first part of the key after being split by the delimiter, and the keyPost refers to the second part of the key after being split by the delimiter.

[0010] S04: Check if there is a linked list of keyPre1 in the hash table. If there is, scan the value linked list to see if it matches. If it matches, notify the corresponding listening client. If it does not match, do not notify any client. If there is no linked list of keyPre1 in the hash table, check if there is a delimiter in keyPre1.

[0011] S05: If there is no delimiter, the process ends. If there is a delimiter, the keyPre1 is split using the delimiter, and S04 is repeated until there is no delimiter in the key.

[0012] Furthermore, the subscription relationship of the key described in S01 is stored using a hash and linked list method.

[0013] Furthermore, if the key mentioned in S02 is a module, type, or instance, then the keyword is empty; if the key is a module, type, instance, or keyword, then the keyword exists.

[0014] Furthermore, as described in S04, the value chain is scanned to check for a match. If the keyword is empty or the keyword in the value chain is a prefix of keyPost, then a match is found.

[0015] In a second aspect of the invention, an apparatus for optimizing fuzzy matching in a large-scale monitoring of a KV database is provided. The apparatus includes:

[0016] Segmentation module: Used to specify the delimiter for the monitored key, and to segment each key by module, type, instance, and keyword using this delimiter;

[0017] Listening module: When a client listens for a key, it finds the last delimiter, calculates the hash value of the string, and saves the linked list corresponding to the module, type, and instance to the hash table. The value of the hash table is a linked list used to store the key.

[0018] The splitting module is used when updating a key in the database. It first finds the last delimiter in the key and splits the key into keyPre1 and keyPost using the delimiter. keyPre1 refers to the first part of the key after splitting by the delimiter, and keyPost refers to the second part of the key after splitting by the delimiter.

[0019] The lookup module is used to check if a linked list of keyPre1 exists in the hash table. If it does, it scans the value linked list to see if it matches. If it matches, it notifies the corresponding listening client. If it does not match, it does not notify any client. If there is no linked list of keyPre1 in the hash table, it checks if there is a delimiter in keyPre1.

[0020] Loop module: If there is no delimiter, the process ends; if there is a delimiter, the keyPre1 is split using the delimiter, and S04 is repeated until no delimiter exists in the key.

[0021] Furthermore, the subscription relationship of the key mentioned in the segmentation module is stored using a hash and linked list method.

[0022] Furthermore, if the key in the segmentation module is module, type, or instance, then the keyword is empty; if the key is module, type, instance, or keyword, then the keyword exists.

[0023] Furthermore, the search module scans the value chain to check for a match. If the keyword is empty or the keyword in the value chain is a prefix of keyPost, then a match is found.

[0024] The above-mentioned English abbreviations have the following meanings:

[0025] KV: Key-Value, a type of database that stores data in key-value pairs.

[0026] key: A unique identifier for the data.

[0027] value: The data content corresponding to the key.

[0028] Hash: A hash function transforms an input of arbitrary length into a fixed-length output using a hash algorithm; this output is called the hash value.

[0029] SQL: Structured Query Language, is a database language with multiple functions such as data manipulation and data definition.

[0030] Redis: Remote Dictionary Server, an open-source, high-performance in-memory data storage system. CPU: Central Processing Unit.

[0031] This invention significantly reduces key matching time and CPU usage, and improves the performance of KV databases by splitting the key and storing the listening client in a hash + linked list, especially in cases of large-scale listening.

[0032] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0033] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. Wherein:

[0034] Figure 1 A flowchart of a method for optimizing fuzzy matching for a large number of KV database listeners according to an embodiment of the present invention is shown;

[0035] Figure 2 A block diagram of an apparatus for optimizing fuzzy matching of a large number of KV database listeners according to an embodiment of the present invention is shown. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] According to an embodiment of the present invention, a method and apparatus for optimizing fuzzy matching for large-scale monitoring of KV databases are proposed. By splitting the key and saving the monitoring client to a hash + linked list, the key matching time can be greatly reduced, CPU usage can be reduced, and the performance of the KV database can be improved under large-scale monitoring.

[0038] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0039] Figure 1 This is a schematic flowchart illustrating a method for optimizing fuzzy matching in a large-scale KV database monitoring process according to an embodiment of the present invention. The method includes:

[0040] S01: Specifies the delimiter for the listening key, and uses this delimiter to segment the module, type, instance, and keyword of each key;

[0041] S02: When the client listens for the key, it finds the last delimiter, calculates the hash value of the string, and saves the linked lists corresponding to the module, type, and instance to the hash table. The value of the hash table is a linked list used to store the key.

[0042] S03: When updating a key in the database, first find the last delimiter in the key, and use the delimiter to split the key into: keyPre1 and keyPost. The keyPre1 refers to the first part of the key after being split by the delimiter, and the keyPost refers to the second part of the key after being split by the delimiter.

[0043] S04: Check if there is a linked list of keyPre1 in the hash table. If there is, scan the value linked list to see if it matches. If it matches, notify the corresponding listening client. If it does not match, do not notify any client. If there is no linked list of keyPre1 in the hash table, check if there is a delimiter in keyPre1.

[0044] S05: If there is no delimiter, the process ends. If there is a delimiter, the keyPre1 is split using the delimiter, and S04 is repeated until there is no delimiter in the key.

[0045] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0046] To provide a clearer explanation of the fuzzy matching optimization method for large-scale monitoring of the KV database, two specific embodiments are described below. However, it is worth noting that these embodiments are only for better illustrating the present invention and do not constitute an improper limitation of the present invention.

[0047] The following two specific examples will further illustrate the method for optimizing fuzzy matching for large-scale monitoring of KV databases:

[0048] Example 1:

[0049] The database uses " / " as the separator for the listening key, and segments the key's module, type, instance, and keyword using this separator. Client A connects to the database and listens on the string / module / httpStack / hs_web1 / host.

[0050] When the database receives the listening string, it finds the last " / ", calculates the hash value of " / module / httpStack / hs_web1", and stores the linked list corresponding to / module / httpStack / hs_web1 / in the hash table. The first data in the linked list is the host.

[0051] The database received an update request from a client, updating the string " / module / httpStack / hs_web1 / hostname" to "wap1".

[0052] Find the " / " at the end of the key, split the key into: / module / httpStack / hs_web1 / hostname, extract / module / httpStack / hs_web1 / and hostname. At this time, / module / httpStack / hs_web1 / is keyPre1, and hostname is keyPost.

[0053] The database searches the hash table to find the listener list corresponding to / module / httpStack / hs_web1 / . It retrieves the first data, host, from the listener list. Since host is a prefix of hostname, it matches. Therefore, client A needs to be notified that the data / module / httpStack / hs_web1 / hostname has been updated to wap1.

[0054] Continue checking if there is any data left in the linked list. Since there is no data left, continue searching if there is still a " / " in the split keyPre1. If there is a " / ", split it into: / module / httpStack / and hs_web1 / hostname. At this time, / module / httpStack / is keyPre1 and hs_web1 / hostname is keyPost.

[0055] If the database searches the hash table and the corresponding listener list for / module / httpStack / is not found, then there is no match and no client needs to be notified.

[0056] Continue checking if there is any data left in the linked list. Since there is no data left, continue splitting forward from / module / httpStack / hs_web1 / hostname. Next, continue checking if there is still a " / " in the split keyPre1. If there is a " / ", split it into: / module / and httpStack / hs_web1 / hostname. At this time, / module / is keyPre1, and httpStack / hs_web1 / hostname is keyPost.

[0057] If the database searches the hash table and the corresponding listener list for / module / is not found, then there is no match and no client needs to be notified.

[0058] Continue checking if there is any data left in the linked list. Since there is no data left, continue splitting forward from / module / httpStack / hs_web1 / hostname. Next, continue searching if there is still a " / " in the split keyPre1. If there is a " / ", split it into: / and module / httpStack / hs_web1 / hostname. At this time, / is keyPre1, and module / httpStack / hs_web1 / hostname is keyPost.

[0059] If the database searches the listener table and the corresponding listener list is not found, then there is no match and no client needs to be notified.

[0060] Continue checking if there is any data left in the linked list. Since there is no data left, continue cutting forward to / module / httpStack / hs_web1 / hostname. Next, continue checking if there is any " / " in the split keyPre1. If there is no " / ", the listening process ends.

[0061] Example 2:

[0062] The database uses " / " as the separator for the listening key, and the key's module, type, instance, and keyword are segmented using this separator. Client B connects to the database and listens for the string " / module / httpStack / ".

[0063] The database receives the listening string, finds the last " / ", calculates the hash value of " / module / httpStack / ", and stores the linked list corresponding to " / module / httpStack / " in the hash table. The first data in the linked list is empty.

[0064] The database received an update request from a client, updating the string " / module / httpStack / hs_web1 / hostname" to "wap1".

[0065] Find the " / " at the end of the key, split the key into: / module / httpStack / hs_web1 / and hostname. At this time, / module / httpStack / hs_web1 / is keyPre1 and hostname is keyPost;

[0066] If the database searches the hash table and the corresponding listener list for / module / httpStack / hs_web1 / is not found, then there is no match and no client needs to be notified.

[0067] Continue checking if there is any data left in the linked list. Since there is no data left, continue to check if there is still a " / " in the split keyPre1. If there is a " / ", it is split into: / module / httpStack / and hs_web1 / hostname. At this time, / module / httpStack / is keyPre1 and hs_web1 / hostname is keyPost.

[0068] The database searches the hash table to find the listening list corresponding to / module / httpStack / , retrieves the first data in the listening list. Since the first data is empty, it is necessary to notify client B that the data / module / httpStack / hs_web1 / hostname has been updated to wap1.

[0069] Continue checking if there is any data left in the linked list. Since there is no data left, continue to check if there is still a " / " in the split keyPre1. If there is a " / ", split it into: / module / and httpStack / hs_web1 / hostname. At this time, / module / is keyPre1 and httpStack / hs_web1 / hostname is keyPost.

[0070] If the database searches the hash table and the corresponding listener list for / module / is not found, then there is no match and no client needs to be notified.

[0071] Continue checking if there is still data in the linked list. Next, continue searching if there is still " / " in the split keyPre1. If there is " / ", split it into: / and module / httpStack / hs_web1 / hostname. At this time, / is keyPre1 and module / httpStack / hs_web1 / hostname is keyPost.

[0072] If the database searches the listener table and the corresponding listener list is not found, then there is no match and no client needs to be notified.

[0073] Continue checking if there is any data left in the linked list. Since there is no data left, the next step is to check if there is still a " / " in the split keyPre1. If there is no " / ", the listening process ends.

[0074] Based on the same inventive concept, this invention also proposes a device for optimizing fuzzy matching in a large-scale KV database monitoring operation. The implementation of this device is similar to the implementation of the method described above, and repeated details will not be elaborated further. Figure 2 As shown, the device 100 includes:

[0075] Segmentation module 101: Used to specify the delimiter for the listening key, and to segment each key by module, type, instance, and keyword using this delimiter;

[0076] Listening module 102: When the client listens for a key, it finds the last delimiter, calculates the hash value of the string, and saves the linked list corresponding to the module, type, and instance to the hash table. The value of the hash table is a linked list used to store the key.

[0077] Segmentation module 103: When updating a key in the database, it first finds the last delimiter in the key and uses the delimiter to split the key into keyPre1 and keyPost. keyPre1 refers to the first part of the key after being split by the delimiter, and keyPost refers to the second part of the key after being split by the delimiter.

[0078] Search module 104: Used to search if there is a linked list of keyPre1 in the hash table. If there is, scan the value linked list to see if it matches. If it matches, notify the corresponding listening client. If it does not match, do not notify any client. If there is no linked list of keyPre1 in the hash table, check if there is a delimiter in keyPre1.

[0079] Loop module 105: If there is no delimiter, the process ends; if there is a delimiter, the keyPre1 is split using the delimiter, and S04 is repeated until no delimiter exists in the key.

[0080] This invention proposes a device for optimizing fuzzy matching in a large-scale KV database monitoring scenario. By splitting the key and storing the monitoring client in a hash + linked list, the device can significantly reduce key matching time, reduce CPU usage, and improve the performance of the KV database under large-scale monitoring conditions.

[0081] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0082] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A method for optimizing fuzzy matching in a large-scale KV database monitoring scenario, characterized in that, The method includes: S01: Specifies the delimiter for the listening key, and segments each key by its module, type, instance, and keyword using this delimiter; S02: When the client listens for the key, it finds the last delimiter, calculates the hash value of the string, and saves the linked lists corresponding to the module, type, and instance to the hash table. The value of the hash table is a linked list used to store the key. S03: When updating a key in the database, first find the last delimiter in the key, and use the delimiter to split the key into: keyPre1 and keyPost. The keyPre1 refers to the first part of the key after being split by the delimiter, and the keyPost refers to the second part of the key after being split by the delimiter. S04: Check if a linked list of keyPre1 exists in the hash table. If it does, scan the value linked list to see if it matches. If the key is empty or the key in the value linked list is a prefix of keyPost, then it matches. If it matches, notify the corresponding listening client. If it does not match, do not notify any client. If there is no linked list of keyPre1 in the hash table, check if there is a delimiter in keyPre1. S05: If there is no delimiter, the process ends. If there is a delimiter, the keyPre1 is split using the delimiter, and S04 is repeated until there is no delimiter in the key.

2. The method for optimizing fuzzy matching in a large-scale KV database according to claim 1, characterized in that, The subscription relationship of the key described in S01 is stored using a hash and linked list method.

3. The method for optimizing fuzzy matching in a large-scale KV database according to claim 1, characterized in that, If the key mentioned in S02 is a module, type, or instance, then the keyword is empty; if the key is a module, type, instance, or keyword, then the keyword exists.

4. A device for optimizing fuzzy matching in a large-scale KV database monitoring operation, characterized in that, The device includes: Segmentation module: Used to specify the delimiter for the monitored key, and to segment each key by module, type, instance, and keyword using this delimiter; Listening module: When a client listens for a key, it finds the last delimiter, calculates the hash value of the string, and saves the linked list corresponding to the module, type, and instance to the hash table. The value of the hash table is a linked list used to store the key. The splitting module is used when updating a key in the database. It first finds the last delimiter in the key and splits the key into keyPre1 and keyPost using the delimiter. keyPre1 refers to the first part of the key after splitting by the delimiter, and keyPost refers to the second part of the key after splitting by the delimiter. The lookup module is used to check if a linked list of keyPre1 exists in the hash table. If it does, it scans the value linked list to check for a match. If the key is empty or the key in the value linked list is a prefix of keyPost, then a match is found. If a match is found, the corresponding listening client is notified; if no match is found, no client is notified. If there is no linked list of keyPre1 in the hash table, it checks if there is a delimiter in keyPre1. Loop module: If there is no delimiter, the process ends. If there is a delimiter, the keyPre1 is split using the delimiter, and S04 is repeated until there is no delimiter in the key.

5. The apparatus for optimizing fuzzy matching in a large-scale KV database monitoring operation according to claim 4, characterized in that, The subscription relationship of the key mentioned in the segmentation module is stored using a hash and linked list method.

6. The apparatus for optimizing fuzzy matching in a large-scale KV database monitoring operation according to claim 4, characterized in that, If the key mentioned in the segmentation module is module, type, or instance, then the keyword is empty; if the key is module, type, instance, or keyword, then the keyword exists.

Images