Memory multi-thread fuzzy retrieval method

Abstract

The invention discloses a memory multi-thread fuzzy retrieval method. The method comprises the following steps: respectively calculating an editing distance similarity value and a common sequence similarity value of an entry and a user retrieval word by adopting a multi-thread parallel operation mode; and calculating a mean value of the editing distance similarity value and the common sequence similarity value corresponding to each entry and the search word, sorting the mean values, and selecting the first N character strings from the mean values as fuzzy search results. According to the method, under the condition of not depending on a big data retrieval component, a reliable result can be efficiently searched from million-level or even ten million-level entries in an extremely short time according to the retrieval word input by the user.

Description

Technical Field

[0001] The present invention relates to the technical field of fuzzy retrieval, and in particular to a memory multi-threaded fuzzy retrieval method. Background Art

[0002] Fuzzy search is a method of searching by using string similarity values. It solves the limitation of exact match search and greatly improves the recall rate of search results. Fuzzy search is often used in e-commerce product name search, encyclopedia knowledge search and other scenarios. Through fuzzy search technology, it can obtain good search results even for typos and inaccurate search terms in user search terms.

[0003] Current data fuzzy retrieval mainly includes database fuzzy matching, search engine full-text retrieval, etc.

[0004] Data retrieval includes data screening and data sorting. In database retrieval, common databases such as MySQL, SqlServer, Oracle, etc. mainly use indexing to search and sort similar data based on the user's search terms and sorting conditions; in search engine retrieval, current mainstream search engines such as Solr and ElasticSearch use full-text retrieval plus TF / IDF inverted indexing to achieve data retrieval and sorting functions.

[0005] When the amount of data stored in a database reaches millions or even tens of millions, the indexes created by the database will take up a lot of disk space, and the retrieval efficiency will become slower and slower as the amount of data increases. The use of search engine technology requires the maintenance of an additional set of search engine software, and search engines are often deployed in a distributed manner, which also has certain requirements for hardware resources. Summary of the invention

[0006] In view of the problems of slow traditional database retrieval, high search engine hardware requirements and high maintenance costs, the present invention provides a memory multi-threaded fuzzy retrieval method that can achieve efficient and fast data retrieval and has high application value.

[0007] The present invention discloses a memory multi-threaded fuzzy retrieval method, which comprises:

[0008] Step 1: Calculate the edit distance similarity value and common sequence similarity value between the entry and the user's search term respectively by multi-threaded parallel computing;

[0009] Step 2: Calculate the average of the edit distance similarity value and the common sequence similarity value corresponding to each entry and the search term and sort them, and select the first N strings as the fuzzy search results.

[0010] Furthermore, the step 1 comprises:

[0011] Obtain entry data and divide the entry data into M parts to form M entry queues;

[0012] According to the search terms input by the user, M thread pools are applied for, each thread pool corresponds to a term queue, and the longest common subsequence length and common subsequence similarity value between each term in the term queue of each thread pool and the search term are calculated; the edit distance and edit distance similarity value between each term in the term queue of each thread pool and the search term are calculated.

[0013] Furthermore, a dynamic programming method is used to calculate the longest common subsequence between the entry and the search term, and the specific steps are:

[0014] Step 101: construct a two-dimensional array result[i][j], which represents the longest common subsequence length of string S1 ending with the i-th character and string S2 ending with the j-th character; string S1 represents the term, and string S2 represents the search term;

[0015] Step 102: Initialize each element in the result array to 0;

[0016] Step 103: Recursively calculate the value of result[i][j]. When S1[i]=S2[j], result[i][j]=

[0017] result[i-1][j-1]+1, otherwise result[i][j]=0; during the recursive process, record the maximum value and index position of result[i][j] to obtain the longest common subsequence and its length.

[0018] Furthermore, according to the longest common subsequence length of each term and the search term and the length of each term and the search term, a common subsequence similarity value of each term and the search term is obtained.

[0019] Furthermore, the calculation formula for the common subsequence similarity value between each term and the search term is:

[0020]

[0021] Among them, sim_lcs(S1,S2) is the common subsequence similarity value between the term S1 and the search term S2, lcs(S1,S2) represents the longest common subsequence length of the two strings S1 and S2, len() represents the length of the string, and the value range of sim_lcs is [0,1].

[0022] Furthermore, the step 2 comprises:

[0023] Calculate the average of the common subsequence similarity value and the edit distance similarity value corresponding to each term in the M term queues, sort the similarity means corresponding to all terms in the M term queues from large to small, and take the top N maximum similarity values ​​as the fuzzy retrieval results.

[0024] Furthermore, the dynamic programming method is used to calculate the edit distance, and the specific steps are as follows:

[0025] Step 201: construct a two-dimensional array edit[i][j], which represents the minimum edit distance between string S1 ending with the i-th character and string S2 ending with the j-th character; string S1 is the entry and string S2 is the search term;

[0026] Step 202: Assign values ​​to the edit[i][j] array according to the following formula;

[0027]

[0028] in,

[0029]

[0030] Step 203: Take the value at the lower right corner of the edit[i][j] array as the edit distance.

[0031] Furthermore, according to the edit distance between each term and the search term and the length of each term and the search term, the edit distance similarity value between each term and the search term is obtained.

[0032] Furthermore, the calculation formula of edit distance similarity is:

[0033]

[0034] Among them, edit(S1,S2) represents the edit distance between the two strings of entry S1 and search term S2, len() represents the length of the string, and the value range of sim_edit is [0,1].

[0035] Furthermore, the step 2 comprises:

[0036] Combine the similarity means of all threads and sort them from large to small. According to the user's search requirements, take out the top N data outputs and release M threads.

[0037] Due to the adoption of the above technical solution, the present invention has the following advantages:

[0038] 1. Compared with the search results of using the edit distance or the longest common subsequence length alone, the present invention combines the sorting of the longest common subsequence length and the sorting of the shortest edit distance, making the final sorting result more reasonable and effective. At the same time, multi-threading is used for parallel calculation, which greatly improves the search efficiency.

[0039] 2. Low hardware and maintenance costs. The retrieval method can be encapsulated into a function and integrated into the project, without the need to maintain additional software, reduce external dependencies, or adopt a big data framework or add servers for distributed deployment.

[0040] 3. Fast retrieval speed. This retrieval method uses multi-threading to perform concurrent calculations, and the calculation results are accurate and fast.

[0041] 4. Good retrieval results. This retrieval method combines the advantages of the longest common subsequence and the edit distance, and solves the shortcomings of the longest common subsequence being sorted only by length and the edit distance being sorted only by distance, making the sorting results more reasonable and the effect better.

[0042] 5. Without relying on big data retrieval components (such as ES, Solr), the present invention can efficiently search for reliable results from millions or even tens of millions of entries in a very short time for the search terms entered by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0044] Figure 1 This is a schematic diagram of the overall process of multi-threaded fuzzy retrieval of the present invention;

[0045] Figure 2 A schematic diagram of the longest common subsequence calculation matrix of the present invention;

[0046] Figure 3 Schematic diagram of the edit distance calculation matrix of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0048] See also Figure 1 The present invention provides an embodiment of a memory multi-threaded fuzzy retrieval method, which includes:

[0049] Step 1: Calculate the edit distance similarity value and common sequence similarity value between the entry and the user's search term respectively by multi-threaded parallel computing;

[0050] Step 2: Calculate the average of the edit distance similarity value and the common sequence similarity value corresponding to each entry and the search term and sort them, and select the first N strings as the fuzzy search results.

[0051] In this embodiment, the step 1 includes:

[0052] Read entry data from a database or file, and divide the entry data into M parts to form M entry queues;

[0053] According to the search terms input by the user, M thread pools are applied for, each thread pool corresponds to a term queue, and the longest common subsequence length and common subsequence similarity value between each term in the term queue of each thread pool and the search term are calculated; the edit distance and edit distance similarity value between each term in the term queue of each thread pool and the search term are calculated.

[0054] In this embodiment, the longest common subsequence is to find the longest discontinuous subsequence between two or more strings. The longest common subsequence is calculated using a dynamic programming method, see Figure 2 , which is the entire process of quickly searching for the common substring "EFED" of substring S1="DEFEGDE" and substring S2="EGFDED".

[0055] The dynamic programming method is used to calculate the longest common subsequence between the entry and the search term. The specific steps are:

[0056] Step 101: construct a two-dimensional array result[i][j], which represents the longest common subsequence length of string S1 ending with the i-th character and string S2 ending with the j-th character; string S1 represents the term, and string S2 represents the search term;

[0057] Step 102: Initialize each element in the result array to 0;

[0058] Step 103: Recursively calculate the value of result[i][j]. When S1[i]=S2[j], result[i][j]=

[0059] result[i-1][j-1]+1, otherwise result[i][j]=0; during the recursive process, record the maximum value and index position of result[i][j] to obtain the longest common subsequence and its length.

[0060] In this embodiment, the common subsequence similarity value between each term and the search term is obtained according to the longest common subsequence length between each term and the search term and the length of each term and the search term.

[0061] In this embodiment, the calculation formula for the common subsequence similarity value between each term and the search term is:

[0062]

[0063] Among them, sim_lcs(S1,S2) is the common subsequence similarity value between the term S1 and the search term S2, lcs(S1,S2) represents the longest common subsequence length of the two strings S1 and S2, len() represents the length of the string, and the value range of sim_lcs is [0,1].

[0064] In this embodiment, step 2 includes:

[0065] Calculate the average of the common subsequence similarity value and the edit distance similarity value corresponding to each term in the M term queues, sort the similarity means corresponding to all terms in the M term queues from large to small, and take the top N maximum similarity values ​​as the fuzzy retrieval results.

[0066] In this embodiment, the edit distance is also called Edit Distance, which refers to the minimum number of operations required to convert one string into another string through insertion, deletion, and replacement operations. The edit distance adopts the dynamic programming method, see Figure 3 , the entire process of calculating the edit distance between substring S1 = "DEFEGDE" and substring S2 = "EGFDED", and the final calculated minimum edit distance is 5.

[0067] The dynamic programming method is used to calculate the edit distance. The specific steps are:

[0068] Step 201: construct a two-dimensional array edit[i][j], which represents the minimum edit distance between string S1 ending with the i-th character and string S2 ending with the j-th character; string S1 is the entry and string S2 is the search term;

[0069] Step 202: Assign values ​​to the edit[i][j] array according to the following formula;

[0070]

[0071] in,

[0072]

[0073] Step 203: Take the value at the lower right corner of the edit[i][j] array as the edit distance.

[0074] In this embodiment, the edit distance similarity value between each entry and the search term is obtained according to the edit distance between each entry and the search term and the length of each entry and the search term.

[0075] In this embodiment, the calculation formula of the edit distance similarity is:

[0076]

[0077] Among them, edit(S1,S2) represents the edit distance between the two strings of entry S1 and search term S2, len() represents the length of the string, and the value range of sim_edit is [0,1].

[0078] In this embodiment, the following formula is used to calculate the average of the common substring similarity value and the edit distance similarity value:

[0079]

[0080] Among them, sim(S1, S2) represents the mean of the common substring similarity value and the edit distance similarity value.

[0081] In this embodiment, step 2 includes:

[0082] Combine the similarity means of all threads and sort them from large to small. According to the user's search requirements, take out the top N data outputs and release M threads.

[0083] Compared with the search results using only the edit distance or the longest common subsequence length, the present invention combines the sorting of the longest common subsequence length and the sorting of the shortest edit distance, making the final sorting result more reasonable and effective. At the same time, multi-threading is used for parallel calculation, which greatly improves the search efficiency.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A memory multi-threaded fuzzy retrieval method, characterized in that: include: Step 1: Calculate the edit distance similarity value and common sequence similarity value between the entry and the user's search term respectively by multi-threaded parallel computing; Step 2: Calculate the average of the edit distance similarity value and the common sequence similarity value corresponding to each entry and the search term and sort them, and select the first N strings as the fuzzy search results.

2. The memory multi-threaded fuzzy retrieval method according to claim 1 is characterized in that: The step 1 comprises: Obtain entry data and divide the entry data into M parts to form M entry queues; According to the search terms input by the user, M thread pools are applied for, each thread pool corresponds to a term queue, and the longest common subsequence length and common subsequence similarity value between each term in the term queue of each thread pool and the search term are calculated; the edit distance and edit distance similarity value between each term in the term queue of each thread pool and the search term are calculated.

3. The memory multi-threaded fuzzy retrieval method according to claim 2 is characterized in that: The dynamic programming method is used to calculate the longest common subsequence between the entry and the search term. The specific steps are: Step 101: construct a two-dimensional array result[i][j], which represents the longest common subsequence length of string S1 ending with the i-th character and string S2 ending with the j-th character; string S1 represents the term, and string S2 represents the search term; Step 102: Initialize each element in the result array to 0; Step 103: Recursively calculate the value of result[i][j]. When S1[i]=S2[j], result[i][j]=result[i-1][j-1]+1, otherwise result[i][j]=0. During the recursive process, record the maximum value and index position of result[i][j] to obtain the longest common subsequence and length value.

4. The memory multi-threaded fuzzy retrieval method according to claim 2 is characterized in that: According to the longest common subsequence length of each entry and the search term and the length of each entry and the search term, the common subsequence similarity value of each entry and the search term is obtained.

5. The memory multi-threaded fuzzy retrieval method according to claim 4 is characterized in that: The calculation formula for the common subsequence similarity value between each term and the search term is: Among them, sim_lcs(S1,S2) is the common subsequence similarity value between the term S1 and the search term S2, lcs(S1,S2) represents the longest common subsequence length of the two strings S1 and S2, len() represents the length of the string, and the value range of sim_lcs is [0,1].

6. The memory multi-threaded fuzzy retrieval method according to claim 1 is characterized in that: The step 2 comprises: Calculate the average of the common subsequence similarity value and the edit distance similarity value corresponding to each term in the M term queues, sort the similarity means corresponding to all terms in the M term queues from large to small, and take the top N maximum similarity values ​​as the fuzzy retrieval results.

7. The memory multi-threaded fuzzy retrieval method according to claim 1 is characterized in that: The dynamic programming method is used to calculate the edit distance. The specific steps are: Step 201: construct a two-dimensional array edit[i][j], which represents the minimum edit distance between string S1 ending with the i-th character and string S2 ending with the j-th character; string S1 is the entry and string S2 is the search term; Step 202: Assign values ​​to the edit[i][j] array according to the following formula; in, Step 203: Take the value at the lower right corner of the edit[i][j] array as the edit distance.

8. The memory multi-threaded fuzzy retrieval method according to claim 1 is characterized in that: According to the edit distance between each term and the search term and the length of each term and the search term, the edit distance similarity value between each term and the search term is obtained.

9. The memory multi-threaded fuzzy retrieval method according to claim 8, characterized in that: The calculation formula of edit distance similarity is: Among them, edit(S1,S2) represents the edit distance between the two strings of entry S1 and search term S2, len() represents the length of the string, and the value range of sim_edit is [0,1].

10. The memory multi-threaded fuzzy retrieval method according to claim 2, characterized in that: The step 2 comprises: Combine the similarity means of all threads and sort them from large to small. According to the user's search requirements, take out the top N data outputs and release M threads.

Images