Method for multi-level dynamic hash storage and matching of domain name and url and storage medium

Abstract

The application relates to the technical field of domain name matching, and discloses a multi-level dynamic hash storage and matching method for domain names and URLs and a storage medium, which comprises multi-level dynamic hash storage logic: creating a hash table, wherein the hash table comprises a first-level host hash table, a first-level URI hash table and a second-level URI hash table; receiving a domain name, searching for a legal host field, a URI field and a URL field, adding the legal host field into the first-level host hash table, adding the legal URI field into the first-level URI hash table, and when the legal URL field exists, splitting the URL field into a host field and a URI field, adding the host field into the first-level host hash table, and adding the URI field into the second-level URI hash table; without increasing the addition limitation of the second-level hash table item and without influence, the use of the memory is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of domain name matching, in particular to a multi-level dynamic hash storage and matching method for domain names and URLs and a storage medium. BACKGROUND

[0002] The current common method for searching domain names and URLs usually adopts hash, skip list and other methods. For example, when using the hash algorithm, a hash table is created for each of the url, host and uri, so that three hash searches are required when performing business matching. The search performance can meet the actual requirements, but the memory occupation is relatively high. The main reason is that the url, host and uri string specifications are relatively long, and creating three hash tables for each of them puts a large pressure on the memory. However, it is known that the url is composed of the host and the uri, that is, the host part of the url can be reused with the host field or the uri part of the url can be reused with the uri field. This can greatly reduce the use of memory, and in the case of large business requirements, it can reduce the pressure on the system memory. In addition, the traditional multi-level hash usually limits the number of second-level hash table entries in consideration of the search performance, and cannot well support the case of a large number of host same uri different urls. At the same time, such algorithms can only provide accurate matching mode, and do not support fuzzy matching required in actual business requirements, which usually uses other methods such as intel's hyperscan, but hyperscan needs to be compiled before use and consumes a long time. SUMMARY

[0003] In view of the deficiencies in the prior art, the purpose of the present application is to provide a multi-level dynamic hash storage and matching method for domain names and URLs and a storage medium.

[0004] In order to achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0005] The multi-level dynamic hash storage and matching method for domain names and URLs comprises a multi-level dynamic hash storage logic:

[0006] A hash table is created, which comprises a first-level host hash table, a first-level uri hash table and a second-level uri hash table;

[0007] A domain name is received, and a legal host field, uri field and url field are sequentially searched,

[0008] When a legal host field exists, the host field is added to the first-level host hash table,

[0009] When there is a legal uri field, the uri field is added to the primary uri hash table,

[0010] When there is a legal url field, the url field is split into a host field and a uri field, wherein the host field is added to the primary host hash table and the uri field is added to the secondary uri hash table.

[0011] In the present application, preferably, before adding the uri field to the secondary uri hash table, the capacity of the secondary uri hash table is determined, and when the capacity of the secondary uri hash table is full, the secondary uri hash table is first expanded, and then the uri field is added to the secondary uri hash table.

[0012] In the present application, preferably, the host field is also searched:

[0013] First, a complete host field search is performed, and when found, the corresponding host field in the hash table is directly returned;

[0014] When not found and there is an entry with a wildcard in the hash table, a fuzzy search is performed, and the search order is to delete the subdomain from front to back according to the domain name.

[0015] In the present application, preferably, the host field is also deleted:

[0016] The host field is searched in the primary host hash table, and the data of the corresponding host field is emptied;

[0017] It is determined whether there is a corresponding uri entry in the secondary uri hash table corresponding to the host field, and when there is no uri entry, the host entry in the primary host hash table is deleted.

[0018] In the present application, preferably, the uri field is also searched:

[0019] First, a complete uri field search is performed, and when found, the corresponding uri field in the hash table is directly returned;

[0020] When not found and there is an entry with a wildcard in the hash table, a fuzzy search is performed, and the search order is to remove the subdirectory from back to front according to the domain name.

[0021] In the present application, preferably, the uri field is also deleted:

[0022] The corresponding uri field is searched, and the corresponding entry is directly deleted.

[0023] In the present application, preferably, the url field is also searched:

[0024] The url field is split into a host field and a uri field, the host field is searched in a first-level host hash table, the uri field part is searched in a second-level uri hash table corresponding to the returned result, and finally the url field search result is returned, and the service data data is returned when the url field is searched.

[0025] In the application, preferably, the deletion of the url field is also included.

[0026] The host field part in the url field is searched in the first-level host hash table, the uri field part is searched in the second-level uri hash table under the corresponding host table item after the host table item is found, and the uri table item is deleted from the second-level uri hash table after it is found.

[0027] Then it is judged whether there are other table items in the second-level uri hash table, when there are none, the second-level uri hash table is recycled, otherwise the second-level uri hash table is retained.

[0028] When the second-level uri hash table is recycled, it is judged whether the service data data corresponding to the host table item is empty, and the corresponding host table item is deleted when the service data data state corresponds to empty, otherwise the host table item is retained.

[0029] In the application, preferably, the dynamic expansion processing of the second-level uri hash table is also included.

[0030] When a second-level table item is added in the second-level uri hash table, it is first judged whether the capacity of the second-level uri hash table reaches a set value.

[0031] When the set value is exceeded, a second-level hash table with large capacity is created.

[0032] When the second-level uri hash table is deleted, it is first judged whether the current second-level uri hash table capacity is less than half of the set value.

[0033] When it is less, a second-level hash table with low capacity is re-applied, and the table items are added from the current second-level uri hash table to the second-level hash table with low capacity.

[0034] Compared with the prior art, the application has the beneficial effects that:

[0035] The method of the present application adopts the multi-level hash mode by disassembling the url into the host and uri combination, the host is the first host hash table, the uri part is the second uri hash table, the host part of the url and the host field are multiplexed, and the second uri hash table is dynamically expanded, so that the memory usage can be greatly reduced without increasing the second hash table item adding limit and without affecting the search performance. The problems of the traditional hash algorithm are effectively solved, and a good solution is provided for business use. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The flowchart of the multi-level dynamic hash storage logic in the multi-level dynamic hash storage and matching method of the domain name and url of the present application.

[0037] Figure 2 The flowchart of the matching process in the multi-level dynamic hash storage and matching method of the domain name and url of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0040] Noun explanation:

[0041] Hash Table, also known as Hash Table, is a data structure used to store and retrieve data. It realizes fast data access by mapping keys to a fixed location (index). URL (Uniform Resource Locator) is a format of string used to identify and locate resources on the Internet. Host domain name is a string used to identify and locate websites on the Internet. It consists of a set of characters arranged according to certain rules, representing the name of one or more websites, servers or other network resources. URI (Uniform Resource Identifier) is a string used to identify and locate resources on the Internet. It is a unified naming scheme used to uniquely identify various resources such as web pages, images, videos, files, etc. Wildcard is a special character used to match a set of patterns in computers. It is widely used in search, file processing and programming to quickly match strings or file names that meet certain conditions. Please see Figure 1 and Figure 2 A preferred embodiment of the present application provides a multi-level dynamic hash storage and matching method for domain names and URLs. By combining the host field and the uri field into a form, using multi-level hash table storage, the host field is stored in the first-level host hash table, and the uri field is stored in the second-level uri hash table. The host part of the url and the host field are reused, and the second-level hash table supports dynamic expansion. Without increasing the table entry addition limit of the second-level hash table and without affecting the search performance, the use of memory can be greatly reduced. At the same time, it supports fuzzy matching of host pre-wildcard, uri post-wildcard and url pre-post-wildcard. Effectively solves the problem of traditional hash algorithm, and provides a good solution for business use.

[0042] The method specifically includes multi-level dynamic hash storage logic:

[0043] Create a hash table, which includes a first-level host hash table, a first-level uri hash table and a second-level uri hash table. When creating a hash table, calculate the required memory of the hash table according to the configuration of the hash table, apply for memory, and initialize the hash structure. By reasonably configuring and initializing the hash table, the foundation for subsequent efficient storage and search operations is laid, which helps to optimize memory usage and improve search performance.

[0044] The domain name is received, and a legal host field, a legal uri field and a legal url field are sequentially searched, wherein the legal host field is that a wildcard "*" can only appear at the first character, and the next character must be "."; the characters in the host field can only be letters, numbers, •, _, -,?, +, [ and ]; the legal uri field is that a wildcard "*" can only appear at the end, and the previous character is " / "; and the legal url field is that a wildcard "*" can only appear at the beginning and end of a string, the character after the first "*" must be ".", and the character before the last "*" must be " / ". The domain name is determined to belong to which type of the three fields according to the legality of the three fields. Through strict legality checking and special wildcard processing, the correctness and consistency of each field are ensured, and the robustness and flexibility of the system are improved.

[0045] When the legal host field exists, the host field is added to the primary host hash table, and when the wildcard exists in the host field, a wildcard mark bit needs to be added in the table entry.

[0046] When the legal uri field exists, the uri field is added to the primary uri hash table, and if the wildcard exists, a wildcard mark bit needs to be added in the table entry.

[0047] When the legal url field exists, the url field is split into a host field and a uri field, wherein the host field is added to the primary host hash table, and the uri field is added to the secondary uri hash table. Through splitting the url into the host and the uri for storage, and dynamically expanding the secondary hash when needed, efficient storage and flexible capacity management are realized, and the problem of storing a large number of host same uri different urls is solved. Through the multi-level hash structure and data reuse, the memory occupation is reduced.

[0048] In the embodiment, before the uri field part is added to the secondary uri hash table, the capacity of the secondary uri hash table is further judged. When the capacity of the secondary uri hash table is full, the secondary uri hash table is first expanded, and then the uri table part is added to the secondary uri hash table. Meanwhile, it is further judged whether there is a secondary uri hash table. When there is no secondary uri hash table, a secondary uri hash table memory is applied from the memory pool, and then the uri table is added.

[0049] In the embodiment, the host field is further searched.

[0050] First, a complete host field search is performed, and when found, the corresponding host field in the hash table is directly returned; when not found and there is a wildcard-containing entry in the hash table, a fuzzy search is performed, and the search order is to remove subdomains from front to back in the domain name, for example, www.abcdefg.com, first search.abcdefg.com, and then search.com, and the matched entry must be marked with a wildcard.

[0051] In the embodiment, host field deletion is also included:

[0052] In the primary host hash table, the host field is searched, the data of the corresponding host field is emptied, the data is used to store user-defined service data, and the data being emptied indicates that the service data corresponding to the entry has been deleted; it is judged whether there is a corresponding uri entry in the secondary uri hash table under the host field, and when there is no uri entry, the host entry in the primary host hash table is deleted. Through the strategy of first accurate search and then fuzzy search, combined with the method of removing subdomains from front to back, the search efficiency is improved, flexible fuzzy matching is supported, and the applicability of the system is enhanced.

[0053] In the embodiment, uri field search is also included:

[0054] First, a complete uri field search is performed, that is, a wildcard "*" appears at the end, and the previous character is " / ", and when found, the corresponding uri field in the hash table is directly returned; when not found and there is a wildcard-containing entry in the hash table, a fuzzy search is performed, and the search order is to remove subdirectories from back to front in the domain name, for example, / home / index.html, first search / home / , and then search / . The matched entry must contain a wildcard. Through the strategy of first accurate search and then fuzzy search, combined with the method of removing subdirectories from back to front, the search efficiency is improved, flexible fuzzy matching is supported, and the adaptability of the system to different uri patterns is enhanced.

[0055] In the embodiment, uri field deletion is also included:

[0056] The corresponding uri field is searched, and the corresponding entry is directly deleted.

[0057] In the embodiment, url field search is also included:

[0058] The url field is split into host field and uri field two parts, the host field is first searched in the primary host hash table, the uri field part is searched in the secondary uri hash table returned by the result, and finally the url field search result is returned, and when the url field is found, the service data data is also returned.

[0059] In the embodiment, the url field is also deleted.

[0060] First, the host field part in the url field is searched in the first-level host hash table, and then the uri field part is searched in the second-level uri hash table under the host table item. After the uri table item is found, it is deleted from the second-level uri hash table. Then, it is determined whether there are other table items in the second-level uri hash table. When there are no other table items, the second-level uri hash table is recycled, otherwise, the second-level uri hash table is retained. When the second-level uri hash table is recycled, it is determined whether the business data data of the corresponding host table item is empty. When the business data data is empty, the corresponding host table item is deleted, otherwise, the host table item is retained. The business data data can be marked by a flag when the business data is not needed to be stored, which indicates whether the current host table item is added by the host field. Meanwhile, the flag is set to 1 when the host field is added, and the host field is not set to 1 when the url field is added, which indicates that the host table item is added by the host field.

[0061] In the embodiment, the second-level uri hash table is also dynamically expanded.

[0062] In order not to affect the search performance in the case of storing a large number of hosts with the same uri and different urls, the second-level uri hash is dynamically expanded. When a second-level table item is added in the second-level uri hash table, it is first determined whether the capacity of the second-level uri hash table reaches a set value. When the capacity exceeds the set value, a second-level hash table with a large capacity is created, and the table items are added from the original second-level uri hash table to the second-level hash table with a large capacity.

[0063] When the second-level uri hash table is deleted, it is first determined whether the capacity of the current second-level uri hash table is less than half of the set value. When the capacity is less than half of the set value, a second-level hash table with a low capacity is re-applied, and the table items are added from the current second-level uri hash table to the second-level hash table with a low capacity. Through dynamic expansion, the problem of the limitation of the number of second-level hash table items in the traditional multi-level hash is solved. By setting the expansion and contraction thresholds, the dynamic balance of memory usage is achieved, and the scalability and resource utilization efficiency of the system are improved.

[0064] In the embodiment, the url is divided into the combination of host and uri, and the host field or the uri field is reused, which saves the memory to a certain extent and provides fuzzy matching capability, as follows:

[0065] The host field supports pre-pending wildcards, the host wildcard "*" must occupy a single dot-separated field by itself and must be prepended, e.g., "*.com", " *.abcdefg.com".

[0066] The uri field supports post-pending wildcards, the uri field wildcard "*" must occupy the last field divided by " / " by itself, e.g., " / home / *", " / *".

[0067] The url field supports both pre- and post-pending wildcards, the url field is split into a host field part and a uri field part, the supported cases are as described above.

[0068] In one embodiment, when the following domain names need to be stored:

[0069] https: / / www.example.com / products / item1

[0070] https: / / www.example.com / products / item2

[0071] https: / / subdomain.example.com / about

[0072] https: / / *.example.com / contact

[0073] It is determined that it is a legal url field, it is split, www.example.com, subdomain.example.com, *.example.com are stored in a first-level host hash table, a corresponding second-level uri hash table is established for each host table item, / products / item1, products / item2 are added to the second-level uri hash table corresponding to the www.example.com table item, / about is added to the second-level uri hash table corresponding to the subdomain.example.com table item, and / contact is stored in the second-level uri hash table corresponding to the *.example.com table item.

[0074] When https: / / www.example.com / products / item1 needs to be accurately found, www.example.com is found in the first-level host hash table, and products / item1 is found in the corresponding second-level uri hash table, and a matching result is found and returned.

[0075] When https: / / test.example.com / contact needs to be found, test.example.com is first found in the first-level host hash table, and then a fuzzy search is performed. example.com in the first-level host hash table is matched, and / contact in the second-level uri hash table corresponding to example.com is found. No matching item is found, and a not found result is returned.

[0076] When https: / / www.example.com / products / item3 needs to be found, www.example.com is found in the first-level host hash table, and / products / item3 in the second-level uri hash table corresponding to www.example.com is found. No matching item is found, and if a URI wildcard exists, a fuzzy search is performed. If not, the process ends.

[0077] In some other preferred embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in the above embodiments.

[0078] The functions described above, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0079] The above description is a detailed description of the preferred embodiments of the present application, but the embodiments are not intended to limit the scope of the patent application of the present application. Any equivalent changes or modifications made under the technical spirit of the present application should be included in the scope of the patent application of the present application.

Claims

1. A method for multi-level dynamic hash storage and matching of domain names and URLs, characterized in that, Includes multi-level dynamic hash storage logic: Create a hash table, which includes a first-level host hash table, a first-level URI hash table, and a second-level URI hash table; Upon receiving the domain name, sequentially search for valid host, uri, and url fields. If a valid host field exists, add the host field to the first-level host hash table. If a valid URI field exists, add the URI field to the first-level URI hash table. If a valid URL field exists, split the URL field into two parts: a host field and a URI field. The host field is added to the first-level host hash table, and the URI field is added to the second-level URI hash table. Before adding the URI field to the secondary URI hash table, the capacity of the secondary URI hash table is checked. If the capacity of the secondary URI hash table is full, the secondary URI hash table is expanded first, and then the URI field is added to the secondary URI hash table.

2. The multi-level dynamic hash storage and matching method for domain names and URLs according to claim 1, characterized in that, This also includes looking up the host field: First, perform a full lookup of the host field. When found, directly return the corresponding host field from the hash table. If no entry is found and a wildcard entry exists in the hash table, a fuzzy search is performed, with the search order based on the domain name, deleting subdomains sequentially from front to back.

3. The multi-level dynamic hash storage and matching method for domain names and URLs according to claim 1, characterized in that, This also includes the deletion of the host field: Look up the host field in the first-level host hash table and set the corresponding host field's data to null; Check if there is a corresponding URI entry in the second-level URI hash table under the host field. If there is no URI entry, delete the host entry in the first-level host hash table.

4. The multi-level dynamic hash storage and matching method for domain names and URLs according to claim 1, characterized in that, This also includes lookups using the URI field: First, perform a full URI field lookup. When found, directly return the corresponding URI field from the hash table. If no entry is found and a wildcard entry exists in the hash table, a fuzzy search is performed, with the search order based on the domain name, removing subdirectories from back to front.

5. The multi-level dynamic hash storage and matching method for domain names and URLs according to claim 1, characterized in that, This also includes the deletion of the uri field: Find the corresponding URI field and delete the corresponding table entry directly.

6. The multi-level dynamic hash storage and matching method for domain names and URLs according to claim 1, characterized in that, It also includes lookups for the url field: The URL field is split into two parts: the host field and the uri field. First, the host field is searched in the first-level host hash table, and then the uri field is searched in the second-level uri hash table corresponding to the returned result. Finally, the URL field search result is returned. When the URL field is found, the business data is also returned.

7. The multi-level dynamic hash storage and matching method for domain names and URLs according to claim 6, characterized in that, This also includes the deletion of the url field: First, search for the host field in the URL field in the first-level host hash table. After finding the corresponding host entry, search for the uri field in the second-level uri hash table under that host entry. Once found, delete the uri entry from the second-level uri hash table. Next, determine if there are any other entries in the secondary URI hash table. If not, reclaim the secondary URI hash table; otherwise, retain the secondary URI hash table. After the secondary URI hash table is recycled, it is determined whether the business data (data) of the corresponding host table entry is empty. If the business data (data) is empty, the corresponding host table entry is deleted; otherwise, the host table entry is retained.

8. The multi-level dynamic hash storage and matching method for domain names and URLs according to claim 6, characterized in that, This also includes dynamic resizing of the secondary URI hash table: When adding a second-level entry to the second-level URI hash table, first check whether the capacity of the second-level URI hash table has reached the set value. When the set value is exceeded, a larger-capacity secondary hash table is created. When deleting a secondary URI hash table, first check if the current capacity of the secondary URI hash table is less than half of the set value; If the number of entries is less than the required amount, a new low-capacity secondary hash table is requested, and the entries are then added from the current secondary URI hash table to the low-capacity secondary hash table.

9. A storage medium, characterized in that, The system contains a computer program that, when executed by a processor, causes the processor to perform the steps of the multi-level dynamic hash storage and matching method for domain names and URLs as described in any one of claims 1-8.

Images