A discrete efficient fuzzy search method and system based on hash table

Abstract

The present invention provides a discrete and efficient fuzzy search method based on a hash table. The method comprises the following steps: step S1, loading source data, performing word segmentation and filtering on the source data; step S2, establishing a hash table; step S3, persisting and saving the hash table; step S4, loading the hash table of the source data; step S5, a user inputs search content, performing word segmentation and filtering on the search content; step S6, inserting synonyms or near synonyms of each word after word segmentation; step S7, searching for each searched word in the hash table of the source data, and counting the frequency and position set of each word; step S8, calculating the matching degree according to the frequency and position of the word, and sorting and outputting the results according to the matching degree. The present invention can perform similarity matching with high efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a discrete high-efficiency fuzzy search method and system based on a hash table. Background Art

[0002] Most of the currently used search systems are based on regular expressions or direct string comparison (exact match). Regular expressions can perform fuzzy matching, but are inefficient and unsuitable for big data and real-time search systems. Direct string matching, while more efficient, cannot perform fuzzy matching or similarity matching. Summary of the Invention

[0003] In order to overcome the above problems, the purpose of the present invention is to provide a discrete and efficient fuzzy search method based on a hash table that can perform similarity matching and has high efficiency.

[0004] The present invention is implemented by the following scheme: a discrete efficient fuzzy search method based on a hash table, the method comprising the following steps:

[0005] Step S1: Load source data, perform word segmentation and filtering on the source data;

[0006] Step S2: Create a hash table;

[0007] Step S3: save the hash table persistently;

[0008] Step S4, loading the hash table of source data;

[0009] Step S5: The user inputs the search content, and the search content is segmented and filtered;

[0010] Step S6: After word segmentation, insert synonyms or near synonyms of each word;

[0011] Step S7: Search each searched word in the hash table of the source data, and count the frequency and position set of each word;

[0012] Step S8: Calculate the matching degree according to the frequency and position of the word, sort and output the results according to the matching degree.

[0013] Furthermore, the step S1 is further specified as follows: loading source data from the database, which includes a data ID and a data content field, segmenting the content field of the source data using a word segmentation algorithm, filtering some non-keywords according to parts of speech, and obtaining a word segmentation list for subsequent further processing.

[0014] Furthermore, the step S2 is further specifically as follows: calculating the hash value of each word, counting the position of each word in the source data, generating a position list, and constructing a hash table.

[0015] Furthermore, the step S3 is further specifically as follows: converting the hash table into a binary data stream and saving the file to facilitate subsequent direct loading and use.

[0016] Furthermore, the step S4 is further specifically as follows: loading a hash table of source data from a previously saved file.

[0017] Furthermore, the step S5 is further specified as follows: the user inputs the content to be searched into the hash table, performs word segmentation on the searched content and filters non-key words, phrases and punctuation marks to generate a word segmentation list.

[0018] Furthermore, the step S6 is further specified as follows: downloading an extended synonym table, constructing a two-dimensional array, each row of the two-dimensional array being a synonym list, searching for corresponding synonyms or near synonyms for each segmented word in the synonym list, and inserting the synonyms into the segmented word list.

[0019] Furthermore, the step S7 is further specified as follows: traversing the word list, calculating a hash value for each word, directly locating the corresponding slot through the hash value, and finding the corresponding source data ID and the location information of the word in the source data.

[0020] Furthermore, the step S8 is further specifically as follows: calculating the matching score according to the position dispersion degree of the word segmentation, and the calculation formula is as follows: Among them: n: the length of the word list to be searched; max: maximum value function; k: constant, here equal to 5; min: minimum value function; x i : The position list of the i-th word segment; the matching score is calculated by the dimension of word segmentation frequency, and the calculation formula is as follows: Among them: max: maximum value function; len(X i ):The length of the position list of the i-th word, that is, the number of times the i-th word appears; k: constant, the same as the k in the position discreteness formula; the matching score is converted into similarity, and the calculation formula is as follows: The search results are sorted by the similarity value of sim and output to complete the search process.

[0021] The present invention also provides a discrete and efficient fuzzy search system based on a hash table, comprising a data processing module, an establishment module, a storage module, a loading module, a search module, an insertion module, a statistics module and an output module; the data processing module is used to load source data and perform word segmentation and filtering on the source data; the establishment module is used to establish a hash table; the storage module is used to persist the hash table; the loading module is used to load the hash table of the source data; the search module is used to perform word segmentation and filtering on the search content when the user inputs the search content; the insertion module is used to insert synonyms or near-synonyms of each word after word segmentation; the statistics module is used to search for each searched word in the hash table of the source data and to count the frequency and position set of each word; the output module is used to calculate the matching degree according to the frequency and position of the word, and to sort and output the results according to the matching degree.

[0022] Furthermore, the data processing module is further specified as follows: loading source data from the database, which includes data ID and data content fields, segmenting the content field of the source data through a word segmentation algorithm, filtering some non-keywords according to parts of speech, and obtaining a word segmentation list for subsequent further processing.

[0023] Furthermore, the establishment module is further specifically configured to: calculate the hash value of each word, count the position of each word in the source data, generate a position list, and construct a hash table.

[0024] Furthermore, the saving module is further specifically configured to convert the hash table into a binary data stream and save the file to facilitate subsequent direct loading and use.

[0025] Furthermore, the loading module is further specifically configured to load a hash table of source data from a previously saved file.

[0026] Furthermore, the search module is further specifically configured as follows: the user inputs the content to be searched into the hash table, performs word segmentation on the searched content and filters non-key words, phrases and punctuation marks to generate a word segmentation list.

[0027] Furthermore, the insertion module is further specifically as follows: downloading an extended version of the synonym table, constructing a two-dimensional array, each row of the two-dimensional array is a synonym list, searching for corresponding synonyms or near synonyms for each segmented word in the synonym list, and inserting them into the segmented word list.

[0028] Furthermore, the statistical module is further specifically configured to: traverse the word list, calculate the hash value for each word, directly locate the corresponding slot through the hash value, and find the corresponding source data ID and the position information of the word in the source data.

[0029] Furthermore, the output module is further specifically configured to calculate the matching score according to the position discreteness of the word segments, and the calculation formula is as follows: Among them: n: the length of the word list to be searched; max: maximum value function; k: constant, here equal to 5; min: minimum value function; x i : The position list of the i-th word segment; the matching score is calculated by the dimension of word segmentation frequency, and the calculation formula is as follows: Among them: max: maximum value function; len(X i ):The length of the position list of the i-th word, that is, the number of times the i-th word appears; k: constant, the same as the k in the position discreteness formula; the matching score is converted into similarity, and the calculation formula is as follows: The search results are sorted by the similarity value of sim and output to complete the search process.

[0030] The beneficial effects of the present invention are:

[0031] Experimental results of the present invention:

[0032] Hardware environment:

[0033]

[0034] Software Environment

[0035]

[0036] The same data source and the same search object are compared under different algorithms:

[0037] Results of this solution

[0038]

[0039] Regular Expressions

[0040]

[0041]

[0042] From the above data, we can see that although this solution has a relatively large memory usage, it has greatly improved the CPU and search speed. In particular, the search speed is basically 10 to 20 times faster than the regular method. The larger the data volume, the more obvious the advantage of this solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic flow chart of the method of the present invention.

[0044] Figure 2 It is a system principle block diagram of the present invention.

[0045] Figure 3 Schematic diagram of the hash table position list.

[0046] Figure 4 A schematic diagram of the hash table structure.

[0047] Figure 5 A schematic diagram of hash table slots.

[0048] Figure 6 A structural diagram of a hash table example. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the accompanying drawings.

[0050] See also Figure 1 As shown, the present invention provides a discrete efficient fuzzy search method based on a hash table, the method comprising the following steps:

[0051] Step S1: Load source data, perform word segmentation and filtering on the source data;

[0052] Step S2: Create a hash table;

[0053] Step S3: save the hash table persistently;

[0054] Step S4, loading the hash table of source data;

[0055] Step S5: The user inputs the search content, and the search content is segmented and filtered;

[0056] Step S6: After word segmentation, insert synonyms or near synonyms of each word;

[0057] Step S7: Search each searched word in the hash table of the source data, and count the frequency and position set of each word;

[0058] Step S8: Calculate the matching degree according to the frequency and position of the word, sort and output the results according to the matching degree.

[0059] The present invention will be further described below by a specific embodiment:

[0060] A hash table (also known as a hash table) is a linear table storage structure. It consists of a directly addressable table and a hash function. The hash function h(k) takes the element key k as its argument and returns the element's storage index, providing an efficient search solution.

[0061] Hash function

[0062] h(k)=k mod m

[0063] m is a constant that determines the size of the hash table. It is set according to the amount of data and is generally around 500 to 2000.

[0064] Hash collision resolution - zipper method

[0065] Since the size of a hash table is finite, and the total number of values ​​to be stored is infinite, for any hash function, there will be a situation where two different elements are mapped to the same position. This situation is called a hash collision.

[0066] Solution:

[0067] See also Figure 3 As shown, each position in the hash table is connected to a linked list. When a conflict occurs, the conflicting element will be added to the end of the linked list at that position.

[0068] (1) Sentence participles

[0069] Split a complete sentence or phrase into individual words. Word segmentation is used to increase the granularity of the search and facilitate fuzzy matching and local matching.

[0070] Word segmentation algorithm - dictionary-based method

[0071] According to a certain strategy, the Chinese character string to be analyzed is matched with the entries in a "big machine dictionary". If a string is found in the dictionary, the match is successful.

[0072] (2) Synonym list

[0073] In order to further improve the search accuracy, it is necessary to add synonym matching. Through the dictionary method, a synonym table is established and its synonyms are inserted into the search content.

[0074] (1) Source data entry module

[0075] Source data can be obtained from a database or imported by the user. During initialization, the system needs to load the source data, preprocess it, create a hash table, and store it locally. Source data refers to the target data, where the user searches for the desired data. Source data is in a one-dimensional array format and contains the data being searched.

[0076] Step 1. Load source data

[0077] Load source data from the database, whose table contains data ID and data content fields.

[0078] Step 2. Data preprocessing

[0079] The source data content field is segmented using a word segmentation algorithm. Non-keywords (modal particles and punctuation marks) are filtered out based on parts of speech to obtain a word segmentation list for subsequent processing.

[0080] Step 3. Create a hash table

[0081] See also Figure 4 As shown, the hash value of each word is calculated, the position of each word in the source data is counted, a position list is generated, and a hash table is constructed.

[0082] Hash value: The hash value of each word after tokenization. A position list is a one-dimensional array that stores the position of each word in the corresponding data; so the generation here refers to constructing a series of position data into a position array (or list).

[0083] See also Figure 5 As shown in the figure, slots correspond to the information of each word. The same hash value may contain different words, and different words correspond to different slots.

[0084] KEY: string type, corresponding to the value of the word.

[0085] ID: long integer type, the source data ID corresponding to the word.

[0086] Position list: an integer array that stores the position of the word in the source data.

[0087] null: terminator.

[0088] See also Figure 6 As shown, the present invention is illustrated by way of example:

[0089] The source data are "I love my motherland" and "Go Motherland - China", and the IDs are 0 and 1.

[0090] After word segmentation and filtering, it becomes "I, love, motherland" and "Motherland, China, come on".

[0091] Step 4. Hash table persistence

[0092] Convert the hash table into a binary data stream and save it to a file for subsequent direct loading and use.

[0093] (2) Search module

[0094] The function of the search module is to search the entire hash table according to the content to be queried by the user, calculate the similarity, and sort and return the found results.

[0095] Step 1. Load the hash table

[0096] Load a hash table from a previously saved file.

[0097] Step 2. The user enters what they want to find

[0098] Step 3. Segment and filter the content to be searched

[0099] Same as the previous step, generate a word list

[0100] Step 4. Insert synonyms

[0101] Build a synonym table, find the corresponding synonym for each word segmentation, and insert it into the word segmentation list.

[0102] The method of constructing a synonym table is as follows: downloading the extended version of the synonym table of Harbin Institute of Technology's "Thesaurus"; constructing a two-dimensional array; each row of the two-dimensional array is a synonym list; this two-dimensional data is the "synonym table" mentioned in the article.

[0103] Step 5. Search the hash table

[0104] Traverse the word list, calculate the hash value for each word, directly locate the corresponding slot through the hash value, and find the corresponding source data ID and the location information of the word in the source data.

[0105] Step 6. Calculate similarity based on location information

[0106] (1) The matching score is calculated by the discrete degree of the word position. The calculation formula is as follows:

[0107]

[0108] in:

[0109] n: The length of the word list to be searched.

[0110] max: maximum value function.

[0111] k: constant, here equal to 5.

[0112] min: Minimum function.

[0113] x i : The position list of the i-th word

[0114]

[0115] (2) Calculate the matching score based on the word frequency dimension. The calculation formula is as follows:

[0116] in:

[0117] max: maximum value function.

[0118] len(xi ): The length of the position list of the i-th word, that is, the number of times the i-th word appears (frequency)

[0119] k: constant, same as k in the position discreteness formula

[0120] (3) Convert the matching score to similarity, and the calculation formula is as follows:

[0121]

[0122] Step 7. Sorting the results

[0123] The search results are sorted by sim similarity value and output to complete the search process.

[0124] In summary, the present invention is an efficient fuzzy search system that can search millions of data within 100 to 200 ms; it returns results sorted by matching degree, similar to Baidu search, and the present invention can be applied to message search, content recommendation and other systems; this solution is based on word segmentation + synonyms, synonyms + hash table algorithms, which can not only perform similarity matching, but also have very high efficiency. A single thread can achieve the ability to search millions of data within 100 ms, and can also support returning all results sorted by similarity.

[0125] See also Figure 2 As shown, the present invention also provides a discrete and efficient fuzzy search system based on a hash table, including a data processing module, an establishment module, a storage module, a loading module, a search module, an insertion module, a statistical module and an output module; the data processing module loads the source data and performs word segmentation and filtering on the source data; the establishment module establishes a hash table; the storage module persists the hash table; the loading module loads the hash table of the source data; the search module receives the search content from the user and performs word segmentation and filtering on the search content; the insertion module inserts synonyms or near synonyms of each word after word segmentation; the statistical module searches for each searched word in the hash table of the source data and counts the frequency and position set of each word; the output module calculates the matching degree according to the frequency and position of the word, and sorts and outputs the results according to the matching degree.

[0126] The data processing module is further specifically as follows: loading source data from the database, which includes data ID and data content fields, segmenting the content field of the source data using a word segmentation algorithm, filtering some non-keywords according to parts of speech, and obtaining a word segmentation list for subsequent further processing.

[0127] The establishment module is further specifically as follows: calculating the hash value of each word, counting the position of each word in the source data, generating a position list, and constructing a hash table.

[0128] The saving module is further specifically configured to convert the hash table into a binary data stream and save the file to facilitate subsequent direct loading and use.

[0129] The loading module is further specifically configured to load a hash table of source data from a previously saved file.

[0130] The search module is further specifically as follows: the user inputs the content to be searched in the hash table, performs word segmentation on the searched content and filters non-key words, phrases and punctuation marks to generate a word segmentation list.

[0131] The insertion module is further specifically as follows: downloading an extended synonym table, constructing a two-dimensional array, each line of the two-dimensional array is a synonym list, searching for corresponding synonyms or near synonyms for each segmented word in the synonym list, and inserting the corresponding synonyms or near synonyms into the segmented word list.

[0132] The statistical module is further specifically configured to: traverse the word list, calculate the hash value for each word, directly locate the corresponding slot through the hash value, and find the corresponding source data ID and the position information of the word in the source data.

[0133] The output module is further specifically configured to calculate the matching score according to the position discreteness of the word segments, and the calculation formula is as follows: Among them: n: the length of the word list to be searched; max: maximum value function; k: constant, here equal to 5; min: minimum value function; x i : The position list of the i-th word segment; the matching score is calculated by the dimension of word segmentation frequency, and the calculation formula is as follows: Among them: max: maximum value function; len(X i ):The length of the position list of the i-th word, that is, the number of times the i-th word appears; k: constant, the same as the k in the position discreteness formula; the matching score is converted into similarity, and the calculation formula is as follows: The search results are sorted by the similarity value of sim and output to complete the search process.

[0134] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A discrete efficient fuzzy search method based on hash table, characterized in that: The method comprises the following steps: Step S1: Load source data, perform word segmentation and filtering on the source data; Step S2: Create a hash table; Step S3: save the hash table persistently; Step S4, loading the hash table of source data; Step S5: The user inputs the search content, and the search content is segmented and filtered; Step S6: After word segmentation, insert synonyms or near synonyms of each word; Step S7: Search each searched word in the hash table of the source data, and count the frequency and position set of each word; Step S8: Calculate the matching degree based on the frequency and position of the word, sort and output the results according to the matching degree; The step S2 is further specifically as follows: calculating the hash value of each word, counting the position of each word in the source data, generating a position list, and constructing a hash table; The step S7 is further specifically as follows: traversing the word list, calculating a hash value for each word, directly locating the corresponding slot by the hash value, and finding the corresponding source data ID and the location information of the word in the source data; The step S8 is further specifically as follows: calculating the matching score according to the position dispersion of the word segments, and the calculation formula is as follows: Among them: n: the length of the word list to be searched; max: maximum value function; k: constant, here equal to 5; min: minimum value function; x i : The position list of the i-th word segment; the matching score is calculated by the dimension of word segmentation frequency, and the calculation formula is as follows: Among them: max: maximum value function; len(X i ):The length of the position list of the i-th word, that is, the number of times the i-th word appears; k: constant, the same as the k in the position discreteness formula; the matching score is converted into similarity, and the calculation formula is as follows: The search results are sorted by the similarity value of sim and output to complete the search process.

2. The method of claim 1, wherein: The step S1 is further specifically as follows: loading source data from the database, which includes a data ID and a data content field, segmenting the content field of the source data using a word segmentation algorithm, filtering some non-keywords according to parts of speech, and obtaining a word segmentation list for subsequent further processing.

3. The method of claim 1 , wherein: The step S3 is further specifically as follows: converting the hash table into a binary data stream and saving the file to facilitate subsequent direct loading and use.

4. The method of claim 1 , wherein: The step S4 is further specifically as follows: loading the hash table of the source data from the previously saved file.

5. The method of claim 1 , wherein: The step S5 is further specifically as follows: the user inputs the content to be searched into the hash table, performs word segmentation on the searched content and filters non-key words, phrases and punctuation marks to generate a word segmentation list.

6. The discrete efficient fuzzy search method based on a hash table according to claim 1, characterized in that: The step S6 is further specifically as follows: downloading an extended synonym table, constructing a two-dimensional array, each row of the two-dimensional array is a synonym list, searching for corresponding synonyms or near synonyms for each segmented word in the synonym list, and inserting the corresponding synonyms or near synonyms into the segmented word list.

7. A discrete efficient fuzzy search system based on a hash table, characterized by: The system comprises a data processing module, a creation module, a storage module, a loading module, a search module, an insertion module, a statistics module and an output module; the data processing module is used to load source data, perform word segmentation and filtering on the source data; the creation module is used to create a hash table; the storage module is used to persist the hash table; the loading module is used to load the hash table of the source data; the search module is used to accept search content input by the user, perform word segmentation and filtering on the search content; the insertion module is used to insert synonyms or near-synonyms of each word after word segmentation; the statistics module is used to search for each searched word in the hash table of the source data, and to count the frequency and position set of each word; the output module is used to calculate the matching degree according to the frequency and position of the word, and to sort and output the results according to the matching degree; The establishment module is further specifically configured to: calculate the hash value of each word, count the position of each word in the source data, generate a position list, and construct a hash table; The statistical module is further specifically configured to: traverse the word list, calculate the hash value for each word, directly locate the corresponding slot through the hash value, and find the corresponding source data ID and the location information of the word in the source data; The output module is further specifically configured to calculate the matching score according to the position discreteness of the word segments, and the calculation formula is as follows: Among them: n: the length of the word list to be searched; max: maximum value function; k: constant, here equal to 5; min: minimum value function; x i : The position list of the i-th word segment; the matching score is calculated by the dimension of word segmentation frequency, and the calculation formula is as follows: Among them: max: maximum value function; len(X i ):The length of the position list of the i-th word, that is, the number of times the i-th word appears; k: constant, the same as the k in the position discreteness formula; the matching score is converted into similarity, and the calculation formula is as follows: The search results are sorted by the similarity value of sim and output to complete the search process.

8. The discrete efficient fuzzy search system based on hash table according to claim 7, characterized in that: The data processing module is further specifically as follows: loading source data from the database, which includes data ID and data content fields, segmenting the content field of the source data using a word segmentation algorithm, filtering some non-keywords according to parts of speech, and obtaining a word segmentation list for subsequent further processing.

9. The discrete efficient fuzzy search system based on hash table according to claim 7, characterized in that: The saving module is further specifically configured to convert the hash table into a binary data stream and save the file to facilitate subsequent direct loading and use.

10. The discrete efficient fuzzy search system based on hash table according to claim 7, characterized in that: The loading module is further specifically configured to load a hash table of source data from a previously saved file.

11. The discrete efficient fuzzy search system based on hash table according to claim 7, characterized in that: The search module is further specifically as follows: the user inputs the content to be searched in the hash table, performs word segmentation on the searched content and filters non-key words, phrases and punctuation marks to generate a word segmentation list.

12. The discrete efficient fuzzy search system based on hash table according to claim 7, characterized in that: The insertion module is further specifically as follows: downloading an extended synonym table, constructing a two-dimensional array, each line of the two-dimensional array is a synonym list, searching for corresponding synonyms or near synonyms for each segmented word in the synonym list, and inserting the corresponding synonyms or near synonyms into the segmented word list.

Images