Optimization method, optimization device and storage medium for fuzzy search

Abstract

This application discloses an optimization method, optimization device, and storage medium for fuzzy search. The optimization method includes: acquiring an input query string and multiple document strings stored in a knowledge base; segmenting the query string into multiple query substrings; segmenting any document string from the multiple document strings into multiple document substrings; for each document substring, determining whether the query string contains a target query substring with no fuzzy similarity to the target document substring; if the query string contains a target query substring with no fuzzy similarity to the target document substring, replacing the target query substring with an empty string, and determining the edit distance between the target query substring and the target document substring as a preset value. This application reduces the occurrence of erroneous results returned by fuzzy search and improves the efficiency of fuzzy search.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to an optimization method, optimization device, and storage medium for fuzzy search. Background Technology

[0002] Search engine technology has become one of the important means of acquiring information resources. Search can be simply divided into "fuzzy search" and "precise search." Fuzzy search refers to the search system automatically performing fuzzy retrieval based on the similarity of synonyms or strings entered by the user, thus yielding more search results. Existing fuzzy search technology generally includes the following steps: query, edit distance calculation, sorting, and outputting results. For example, the query string is first divided sequentially according to the length of the strings in the paragraph, resulting in a set of query string substrings. When a string in the paragraph matches a string in the query string, the matching degree of the original string corresponding to that string is added to the length of the original string. If the matching degree of the original string is greater than a preset upper limit and there are no duplicate elements in the position list, the original string is added to the result set; otherwise, the string is verified by edit distance. If the matching degree of the original string is less than a preset lower limit, the original string is directly filtered out. If the matching degree of the original string is between the preset lower and upper limits, the string is verified by edit distance. Existing technologies directly calculate edit distance by querying related substrings, which lacks semantic information. There may be strings with the same characters but completely different meanings. Therefore, while returning results, a large number of inaccurate results are also brought, resulting in a large workload for calculating edit distance and low efficiency of fuzzy search. Summary of the Invention

[0003] The purpose of this application is to provide an optimization method, optimization device, and storage medium for fuzzy search, so as to solve the problem that fuzzy search in the prior art may lead to a large number of inaccurate results and thus low efficiency.

[0004] To achieve the above objectives, the first aspect of this application provides an optimization method for fuzzy search, the optimization method comprising:

[0005] Retrieve the input query string and multiple document strings stored in the knowledge base, respectively;

[0006] Tokenize the query string to obtain multiple query substrings;

[0007] Tokenize any document string from multiple document strings to obtain multiple document substrings from any document string;

[0008] For any document substring, determine whether the query string contains a target query substring that has no fuzzy similarity to the arbitrary document substring;

[0009] If a target query substring exists in the query string and has no fuzzy similarity to any document substring, the target query substring is replaced with an empty string, and the edit distance between the target query substring and any document substring is set to a preset value.

[0010] In the embodiments of this application, multiple query substrings and multiple document substrings of any document string all include word segmentation arrays and part-of-speech arrays, and the part-of-speech arrays and word segmentation arrays correspond one-to-one.

[0011] In this embodiment of the application, determining whether a query string exists as a target query substring with no fuzzy similarity to any document substring includes at least one of the following:

[0012] Based on the word segmentation arrays and part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether the query string contains a query substring that matches the characters of any document substring but has a different part of speech; if the query string contains a query substring that matches the characters of any document substring but has a different part of speech, determine whether the query string contains a target query substring that has no fuzzy similarity to any document substring; or

[0013] Obtain the entity sets of the query string and any document string; based on the word segmentation array, part-of-speech array, and entity set, determine whether the query string contains a substring that matches the characters of any document substring, has the same part of speech but a different entity type; if the query string contains a substring that matches the characters of any document substring, has the same part of speech but a different entity type, determine whether the query string contains a target query substring with no fuzzy similarity to any document substring; or

[0014] Obtain the entity sets of the query string and any document string, as well as the thesaurus of the knowledge base; based on the word segmentation array, part-of-speech array, entity set, and thesaurus, determine whether the query string contains a substring that matches the character set, part of speech, and entity type of any document substring but is not a synonym; if the query string contains a substring that matches the character set, part of speech, and entity type of any document substring but is not a synonym, determine whether the query string contains a target substring with no fuzzy similarity to any document substring.

[0015] In this embodiment of the application, determining whether a query string contains a substring that matches characters of any document substring but has a different part of speech based on the word segmentation array and part-of-speech array of multiple query substrings and multiple document substrings of any document string includes:

[0016] Based on the tokenized arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are characters in the query string that intersect with any document substring;

[0017] If the query string contains characters that intersect with any document substring, retrieve the similar query substring and the similar document substring.

[0018] Based on the part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether similar query substrings and similar document substrings have the same part of speech;

[0019] When similar query substrings and similar document substrings have different parts of speech, it is determined that the query string contains a query substring that matches the characters of any document substring but has a different part of speech.

[0020] In this embodiment of the application, determining whether a query string exists that matches any document substring in terms of characters, part-of-speech tags, but entity types, based on the word segmentation array, part-of-speech tag array, and entity set includes:

[0021] Based on the tokenized arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are characters in the query string that intersect with any document substring;

[0022] If the query string contains characters that intersect with any document substring, retrieve the similar query substring and the similar document substring.

[0023] Based on the part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether similar query substrings and similar document substrings have the same part of speech;

[0024] If the parts of speech of similar query substrings and similar document substrings are the same, determine whether the entity types of similar query substrings and similar document substrings are the same based on the entity set;

[0025] When the entity types of similar query substrings and similar document substrings are different, it is determined that the query string contains a query substring that matches any document substring in terms of characters, has the same part of speech, but has a different entity type.

[0026] In this embodiment of the application, determining whether a query string exists that matches any document substring in terms of character type, part-of-speech tag, entity type, and thesaurus, but is not a synonym, based on the word segmentation array, part-of-speech tag array, entity set, and thesaurus, includes:

[0027] Based on the tokenized arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are characters in the query string that intersect with any document substring;

[0028] If the query string contains characters that intersect with any document substring, retrieve the similar query substring and the similar document substring.

[0029] Based on the part-of-speech arrays of multiple query substrings and multiple document substrings of any document substring, determine whether similar query substrings and similar document substrings have the same part of speech;

[0030] If the parts of speech of similar query substrings and similar document substrings are the same, determine whether the entity types of similar query substrings and similar document substrings are the same based on the entity set;

[0031] If the entity types of similar query substrings and similar document substrings are the same, determine whether the similar query substrings and similar document substrings are synonyms based on the thesaurus;

[0032] If the similar query substring and the similar document substring are not synonyms, it is determined that the query string contains a query substring that matches any document substring in terms of characters, part of speech, and entity type, but is not a synonym.

[0033] In this embodiment of the application, the optimization method further includes:

[0034] Obtain synonyms and specialized terms in the target field;

[0035] A knowledge base is built based on synonyms and specialized terms in the target field.

[0036] In this embodiment of the application, the optimization method further includes:

[0037] Update the knowledge base upon receiving new input synonyms and / or specialized terms.

[0038] A second aspect of this application provides an optimization apparatus for fuzzy search, comprising:

[0039] The memory is configured to store instructions; and

[0040] The processor is configured to retrieve instructions from memory and, when executing instructions, to implement the aforementioned optimization method for fuzzy search.

[0041] A third aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the aforementioned optimization method for fuzzy search.

[0042] The above technical solution involves obtaining the input query string and multiple document strings stored in the knowledge base. The query string and any document strings are then segmented to obtain a query substring and multiple document substrings of the arbitrary document string. For any document substring, if the query string contains a target query substring with no fuzzy similarity to the arbitrary document substring, the target query substring is replaced with an empty string, and the edit distance between the target query substring and the arbitrary document substring is set to a preset value. By replacing query substrings with no fuzzy similarity with empty strings, the number of incorrect results returned by fuzzy search is reduced. Furthermore, setting the edit distance between the target query substring with no fuzzy similarity and the arbitrary document substring to a preset value reduces the calculation of edit distance, thereby improving the efficiency of fuzzy search.

[0043] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0044] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0045] Figure 1 The flowchart illustrating an optimization method for fuzzy search according to an embodiment of this application is shown schematically.

[0046] Figure 2 A flowchart illustrating a method for determining a target query substring according to an embodiment of this application is shown schematically;

[0047] Figure 3 A flowchart illustrating a method for determining a target query substring according to another embodiment of this application is shown schematically;

[0048] Figure 4 A flowchart illustrating a method for determining a target query substring according to yet another embodiment of this application is shown schematically;

[0049] Figure 5 The diagram schematically illustrates a structural block diagram of an optimization apparatus for fuzzy search according to an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0051] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0052] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0053] Figure 1 A flowchart illustrating an optimization method for fuzzy search according to an embodiment of this application is shown schematically. Figure 1 As shown in the figure, this application provides an optimization method for fuzzy search, which may include the following steps.

[0054] Step 101: Obtain the input query string and the multiple document strings stored in the knowledge base.

[0055] In this embodiment, the query string refers to the string received by the processor that needs to be used for fuzzy search, i.e., the query statement. The query string consists of multiple characters. For example, if the user needs to perform a fuzzy search on the sentence "Concrete outlet is normal", the query string received by the processor will be a query string consisting of 7 characters, i.e., "Concrete outlet is normal".

[0056] A knowledge base is a library that pre-stores multiple document strings. A document string is a statement stored in the knowledge base. Multiple document strings refer to multiple statements stored in the knowledge base. A processor can retrieve multiple document strings from the knowledge base to determine if they are fuzzy search results for the query string. For example, the processor can retrieve multiple document strings such as "Concrete output from Line 1" and "Inlet blocked" from the knowledge base to determine if they match the query string "Concrete outlet is normal".

[0057] Step 102: Tokenize the query string to obtain multiple query substrings.

[0058] Step 103: Tokenize any document string from multiple document strings to obtain multiple document substrings from any document string.

[0059] In this embodiment, an arbitrary document string refers to a single document string among multiple document strings. This embodiment compares the query string with each document string individually. For an arbitrary document string, it is necessary to determine whether there is a query substring with no fuzzy similarity to the query string. Therefore, the processor first performs word segmentation on both the query string and the arbitrary document string to obtain multiple query substrings and multiple document substrings of the arbitrary document string. Word segmentation refers to the process by which the processor divides a string into substrings. After word segmentation, the query string yields multiple query substrings, and after word segmentation, the arbitrary document string yields multiple document substrings. This facilitates comparing each query substring with each document substring sequentially to determine whether the arbitrary query substring and the arbitrary document substring are string pairs with no fuzzy similarity.

[0060] Step 104: For any document substring, determine whether there exists a target query substring in the query string that has no fuzzy similarity to the arbitrary document substring.

[0061] In this embodiment, an arbitrary document substring refers to a specific document substring among multiple document substrings. The processor compares each query substring with the arbitrary document substring to determine whether the query substring has no fuzzy similarity to the arbitrary document substring. If it is determined that the query substring has no fuzzy similarity to the arbitrary document substring, it is identified as the target query substring. The target query substring is the query string that has no fuzzy similarity to the arbitrary document substring. In other words, it can be determined that the query string has a target query substring with no fuzzy similarity to the arbitrary document substring. This eliminates the possibility that the target query substring has a similarity relationship with the corresponding arbitrary document substring.

[0062] Step 105: If a target query substring exists in the query string and has no fuzzy similarity to any document substring, replace the target query substring with an empty string, and set the edit distance between the target query substring and any document substring to a preset value.

[0063] In this embodiment, when it is determined that the query string contains a target query substring (i.e., when it is determined that the target query substring has no fuzzy similarity to any document substring currently being compared), the target query substring can be replaced with an empty string, and the edit distance between the current target query substring and any document substring is set to a preset value, i.e., 0. Thus, when calculating the edit distance, the processor only needs to calculate the values ​​of non-empty substring pairs; for substring pairs with no fuzzy similarity, no further calculation is required. This improves the efficiency of edit distance calculation.

[0064] In this embodiment, the input query string and multiple document strings stored in the knowledge base are obtained respectively. Then, the query string and any document strings from the multiple document strings are segmented to obtain query substrings and multiple document substrings from the arbitrary document strings. Next, for any document substring, if the query string contains a target query substring with no fuzzy similarity to the arbitrary document substring, the target query substring is replaced with an empty string, and the edit distance between the target query substring and the arbitrary document substring is determined to a preset value. In this way, by replacing query substrings with no fuzzy similarity with empty strings, the number of incorrect results returned by fuzzy search can be reduced. Furthermore, by determining the edit distance between the target query substring with no fuzzy similarity and the arbitrary document substring to a preset value, the calculation of edit distance is reduced, thereby improving the efficiency of fuzzy search.

[0065] In the embodiments of this application, multiple query substrings and multiple document substrings of any document string all include word segmentation arrays and part-of-speech arrays, and the part-of-speech arrays and word segmentation arrays correspond one-to-one.

[0066] In this embodiment, the processor can compare the query string with each document string individually. For any document string, it is necessary to determine whether there is a query substring with no fuzzy similarity to the query string. Therefore, the processor needs to first perform word segmentation on both the query string and the arbitrary document string to obtain multiple query substrings and multiple document substrings of the arbitrary document string. Word segmentation refers to the process by which the processor divides a string into substrings. After word segmentation, the query string can obtain multiple query substrings, and after word segmentation, the arbitrary document string can obtain multiple document substrings.

[0067] In an embodiment of the present application, after the processor tokenizes the query string and any string, the tokenization result may include a token array and a词性 array. The token array refers to a set of tokens of multiple substrings, and the词性 array refers to a combination of the词性s corresponding to each token of multiple substrings. For example, assume that the query string q1 is "The concrete outlet is normal", and any document string is "Line 1 concrete outlet". The tokenization result of the query string q1 after tokenization is q1 = [[concrete outlet, everything, normal][n, i, v]]. Among them, [concrete outlet, everything, normal] is the token array of multiple query substrings of the query string, and [n, i, v] is the词性 array of multiple query substrings of the query string. Among them, "concrete outlet" is a noun, so the corresponding词性 is n, "normal" is a verb, so the corresponding词性 is v, and so on. Similarly, the tokenization result of any document string T1 is [[Line 1, concrete outlet], [n, v]]. Among them, [Line 1, concrete outlet] is the token array of multiple document substrings, and [n, v] is the词性 array of multiple document substrings.

[0068] In an embodiment of the present application, various tools can be used for tokenization. For example, tokenizing a string may include but is not limited to using the open-source tool Chinese Lexical Analysis (LAC) to tokenize the string. Other tokenization tools can also be used instead.

[0069] In an embodiment of the present application, by tokenizing the string to obtain a substring composed of a token array and a词性 array, it is convenient to compare each query substring with each document substring in turn to determine whether any query substring and any document substring are a pair of strings with non-fuzzy similarity.

[0070] In an embodiment of the present application, in step 104, determining whether there is a target query substring in the query string that has non-fuzzy similarity with any document substring may include at least one of the following:

[0071] According to the token arrays and词性 arrays of multiple query substrings and multiple document substrings of any document string, determine whether there is a query substring in the query string that has character matching but different词性s with any document substring; in the case where there is a query substring in the query string that has character matching but different词性s with any document substring, determine that there is a target query substring in the query string that has non-fuzzy similarity with any document substring; or

[0072] Obtain the entity sets of the query string and any document string; based on the word segmentation array, part-of-speech array, and entity set, determine whether the query string contains a substring that matches the characters of any document substring, has the same part of speech but a different entity type; if the query string contains a substring that matches the characters of any document substring, has the same part of speech but a different entity type, determine whether the query string contains a target query substring with no fuzzy similarity to any document substring; or

[0073] Obtain the entity sets of the query string and any document string, as well as the thesaurus of the knowledge base; based on the word segmentation array, part-of-speech array, entity set, and thesaurus, determine whether the query string contains a substring that matches the character set, part of speech, and entity type of any document substring but is not a synonym; if the query string contains a substring that matches the character set, part of speech, and entity type of any document substring but is not a synonym, determine whether the query string contains a target substring with no fuzzy similarity to any document substring.

[0074] In this embodiment, the processor compares each query substring with any document substring to determine whether the query substring has no fuzzy similarity to any document substring. If it is determined that the query substring has no fuzzy similarity to any document substring, the query substring is identified as the target query substring. The target query substring is the query string that has no fuzzy similarity to any document substring. In other words, it can be determined that the query string contains a target query substring with no fuzzy similarity to any document substring. This eliminates the possibility that the target query substring has a similarity relationship with the corresponding document substring. This embodiment can determine whether the query string contains a target query substring with no fuzzy similarity to any document substring through three cases. As long as at least one of the three cases is satisfied, it indicates that the query string contains a target query substring with no fuzzy similarity to any document substring.

[0075] In one embodiment, the target query substring satisfies condition A. Condition A: The processor can determine whether there exists a query substring in the query string that matches the characters of any document substring but has a different part of speech based on the word segmentation array and part-of-speech array of multiple query substrings and multiple document substrings of any document string; if there exists a query substring in the query string that matches the characters of any document substring but has a different part of speech, it is determined that there exists a target query substring in the query string that has no fuzzy similarity to any document substring.

[0076] Specifically, when a query string contains a similar but different part-of-speech substring compared to any document substring, it is determined that the query string contains a target query substring satisfying condition A. Here, "similar" means that there exists a query substring that shares the same characters as any document substring. The part-of-speech tag can be determined based on the part-of-speech arrays of multiple query substrings after word segmentation and multiple document substrings of any document string.

[0077] Figure 2 The flowchart illustrating a method for determining a target query substring according to an embodiment of this application is shown schematically. In one embodiment of this application, the method for determining the target query substring may include the following steps:

[0078] Step 201: Based on the word segmentation arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are any characters in the query string that intersect with any document substring;

[0079] Step 202: If the query string contains characters that intersect with any document substring, obtain the similar query substring and the similar document substring;

[0080] Step 203: Based on the part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether similar query substrings and similar document substrings have the same part of speech;

[0081] Step 204: If the parts of speech of similar query substrings and similar document substrings are different, determine that there exists a query substring in the query string that matches the characters of any document substring but has a different part of speech;

[0082] Step 205: If the query string contains a query substring that matches the characters of any document substring but has a different part of speech, determine that the query string contains a target query substring that has no fuzzy similarity to any document substring.

[0083] In this embodiment, the processor first needs to determine whether the query string contains a substring similar to any document substring. Therefore, the processor can determine whether the query string contains characters that intersect with any document substring based on the segmentation arrays of multiple query substrings and multiple document substrings of any document string. If so, the query substring and the corresponding document substring are identified as similar query substrings and similar document substrings. For example, the segmentation result of the query string q1 is q1 = [[Concrete outlet, everything, normal][n, i, v]]; the segmentation result of any document string T1 is [[Line 1, concrete outlet]], [n, v]]. That is, the segmentation result of the query string q1 is: concrete outlet n, everything i, and normal v. The segmentation result of any document string T1 is: Line 1 n and concrete outlet v. In one example, the processor can convert the q1 and T1 data structures into objects:

[0084] q1={

[0085] n: [{out, concrete, opening}], # obtained by iterating through the array and then iterating through the string.

[0086] i: [{one, all}],

[0087] v: [{positive, constant}]

[0088] T1 = {

[0089] n: [{Line 1}]

[0090] i: [{out, concrete}]

[0091] }

[0092] Then check if there is a character overlap between words of different parts of speech in T1 for all words in q1, such as:

[0093] U[nv] ij =q1[n] i ∩T1[v] j ==? 0

[0094] U[nn] ij =q1[n] i ∩T1[v] j ==? 0

[0095] If the intersection is U[nv] ij Not equal to 0, U[nn] ij If the value is 0, the corresponding word in q will be deleted during the calculation. The deletion method is to iterate through the array in the word segmentation result q1 and delete the elements.

[0096] It can be obtained through the above method that "concrete outlet" and "concrete discharging" both include "concrete", and "everything" and "Line 1" both include "one". Therefore, it can be determined that "concrete outlet" and "concrete discharging" are similar substring pairs, where "concrete outlet" is the similar query substring and "concrete discharging" is the similar document substring. "Everything" and "Line 1" are similar substring pairs, where "everything" is the similar query substring and "Line 1" is the similar document substring.

[0097] After determining the similar query substring and the similar document substring, it is necessary to determine whether the similar query substring and the similar document substring have the same word nature according to the word nature arrays of multiple query substrings and multiple document substrings of any document string. If they are different, it is determined that the query string has a query substring that matches the characters of any document substring but has a different word nature, that is, the target query substring meets condition A, and the query string has a target query substring that has no fuzzy similarity with any document substring. Still taking q1 and T1 as an example, the corresponding word nature of "concrete outlet" is n, the corresponding word nature of "concrete discharging" is v. Therefore, the word natures of "concrete outlet" and "concrete discharging" are different. The word nature of "everything" is v, and the word nature of "Line 1" is n. Therefore, the word natures of "everything" and "Line 1" are also different. It can be seen that both pairs of substrings meet case A. Therefore, both pairs of substrings have no fuzzy similarity.

[0098] In another embodiment, the target query substring meets case B. Case B: The processor can obtain the entity set of the query string and any document string; determine whether the query string has a query substring that matches the characters, has the same word nature but different entity types of any document substring according to the word segmentation array, the word nature array and the entity set; in the case where the query string has a query substring that matches the characters, has the same word nature but different entity types of any document substring, determine that the query string has a target query substring that has no fuzzy similarity with any document substring.

[0099] Specifically, when the query string and any document substring have similar entities (similar and with the same word nature) but different types of query substrings, it is determined that the query string has a target query substring that meets case B. Among them, the entity can be obtained by identifying the named entity of the string. For example, entity recognition is performed through an entity word list or some machine learning algorithms.

[0100] Figure 3 Schematically shows a flowchart of a method for determining a target query substring according to another embodiment of the present application. As Figure 3 shown, in another embodiment of the present application, the method for determining a target query substring may include the following steps:

[0101] Step 301: Obtain the entity sets of the query string and any document string;

[0102] Step 302: Based on the tokenized arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are characters in the query string that intersect with any document substring;

[0103] Step 303: When there are characters in the query string that intersect with any document substring, obtain the similar query substring and the similar document substring;

[0104] Step 304: Based on the词性 arrays of multiple query substrings and multiple document substrings of any document string, determine whether the similar query substring and the similar document substring have the same词性;

[0105] Step 305: When the similar query substring and the similar document substring have the same词性, determine whether the entity types of the similar query substring and the similar document substring are the same according to the entity set;

[0106] Step 306: When the entity types of the similar query substring and the similar document substring are different, determine that there is a query substring in the query string that has character matching, the same词性 but different entity types with any document substring;

[0107] Step 307: When there is a query substring in the query string that has character matching, the same词性 but different entity types with any document substring, determine that there is a target query substring in the query string that has no fuzzy similarity with any document substring.

[0108] In the embodiment of the present application, steps 302 to 304 can refer to case A. Based on the steps of case A, the processor also needs to first obtain the entity sets of the query string and any document string. For example, the query string q2 is "How many high-altitude operation machines are there", and any document substring T2 is "The computer room is very hot". The tokenized result of the query string q2 is: there v, several m, high-altitude operation n, machine n; the tokenized result of any document substring T2 is: computer room n, very d, hot a. Through entity recognition, it can be obtained that the machine is a part and the computer room is a location. That is:

[0109] q2 = [[{there, several, high-altitude operation, machine}], [v, m, n, n]]

[0110] T2 = [[computer room, very, hot], [n, d, a]].

[0111] Entitydict = {

[0112] Part: [machine,...] It should be noted that the "词性" in the original text is not clearly defined. You may need to clarify or correct this according to the actual situation for a more accurate translation.

[0113] Location: [Computer room,...]

[0114] }

[0115] This application embodiment obtains an object with the following data structure by finding the entities of query string q2 and arbitrary document string T2:

[0116] q2={

[0117] Parts: [{machine, equipment}]

[0118] }

[0119] T2={

[0120] Location: [{computer room}]

[0121] }

[0122] The entities can be obtained through named entity recognition algorithms or through entity vocabulary traversal. The traversal algorithm is not limited, for example, it can be obtained through AC automaton traversal.

[0123] In this embodiment, when the parts of speech of the similar query substring and the similar document substring are the same, the processor determines whether the entity types of the similar query substring and the similar document substring are the same based on the entity set. In one example, the processor can check whether the entities in q2 have character overlap with entities of different entity categories in T2, such as: U[parts, locations]. ij =q2[part] i ∩T2[v] j == 0. If the intersection U[part, location] ij A value not equal to 0 indicates that the entity types of the similar query substring and the similar document substring are different. In other words, if the query string has a target query substring with no fuzzy similarity to any document substring, then the corresponding target query substring in q2 will be deleted during the calculation. The deletion method is to iterate through the array in the word segmentation result q2 and delete the elements.

[0124] In another embodiment, the target query substring satisfies condition C. Condition C: The processor can obtain the entity set of the query string and any document string, as well as the thesaurus of the knowledge base; based on the word segmentation array, part-of-speech array, entity set, and thesaurus, it determines whether the query string has a query substring that matches the characters, has the same part of speech, and has the same entity type as any document substring but is not a synonym; if the query string has a query substring that matches the characters, has the same part of speech, and has the same entity type as any document substring but is not a synonym, it determines that the query string has a target query substring with no fuzzy similarity to any document substring.

[0125] Specifically, when a query string contains entities similar to any document substring, and the entity types are the same, but the query substring is not a synonym of any document substring, it is determined that the query string contains a target query substring that satisfies condition C. The knowledge base in this embodiment may also include a library of professional terms and synonyms related to the current technical field. Taking the field of engineering machinery as an example, the synonyms in the knowledge base may include, but are not limited to: "concrete outlet": ["concrete outlet gate", "unloading port", "discharge gate", "unloading gate"]; "feed inlet": ["feed inlet", "feed gate"]; "level gauge": ["level gauge"]; "thermal overload device": ["thermal overload relay"], etc. The professional terms in the knowledge base may include, but are not limited to: "sand silo", "vibrating motor", "aggregate scale", "screw conveyor", "main machine door", "intermediate silo", "sand and gravel silo", "intermediate compartment door", "aggregate compartment door", etc. The processor can determine whether any query substring and any document substring are synonyms by searching the synonyms.

[0126] Figure 4 A flowchart illustrating a method for determining a target query substring according to yet another embodiment of this application is shown. Figure 4 As shown, in another embodiment of this application, the method for determining the target query substring may include the following steps:

[0127] Step 401: Obtain the entity set of the query string and any document string, as well as the thesaurus of the knowledge base;

[0128] Step 402: Based on the word segmentation arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are any characters in the query string that intersect with any document substring;

[0129] Step 403: If the query string contains characters that intersect with any document substring, obtain the similar query substring and the similar document substring;

[0130] Step 404: Based on the part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether similar query substrings and similar document substrings have the same part of speech;

[0131] Step 405: If the parts of speech of the similar query substring and the similar document substring are the same, determine whether the entity types of the similar query substring and the similar document substring are the same based on the entity set;

[0132] Step 406: If the entity types of the similar query substring and the similar document substring are the same, determine whether the similar query substring and the similar document substring are synonyms based on the thesaurus.

[0133] Step 407: If the similar query substring and the similar document substring are not synonyms, determine that the query string contains a query substring that matches any document substring in terms of characters, part of speech, and entity type, but is not a synonym.

[0134] Step 408: If the query string contains a query substring that matches any document substring in terms of characters, part of speech, and entity type but is not a synonym, determine that the query string contains a target query substring that has no fuzzy similarity to any document substring.

[0135] In this embodiment, steps 402-405 can refer to cases A and B. Based on the steps of cases A and B, the processor also needs to obtain the thesaurus of the knowledge base. Similar to cases A and B, if entities of the same type have character overlap and the two entities are not in the thesaurus combination, the word corresponding to the target query substring is deleted during the calculation. Similarly, a traversal method can be used to determine whether it is in the thesaurus. For example, the query substring q1 is still "Concrete outlet is normal", and the arbitrary document substring T3 is "Inlet is blocked". The word segmentation result of the query substring q1 is: concrete outlet n, everything i, normal v; the word segmentation result of the arbitrary document substring T3 is: inlet n, blocked v, and u1. Although "concrete outlet" and "inlet" are similar entities and have the same entity type, they are not synonyms. Therefore, it can be determined that the substring pair "concrete outlet" and "inlet" satisfies case C, that the string pair has no fuzzy similarity. If a string pair is indeed a synonym but is not currently in the thesaurus, the data needs to be manually added to the thesaurus to update the thesaurus in the knowledge base.

[0136] This application embodiment compensates for the lack of semantic information in fuzzy search in the prior art by excluding the target query substring through case A, case B and / or case C. The target query substring is replaced with an empty string, so that the similarity between the target query substring and the corresponding arbitrary document substring is directly a preset value, i.e., 0. There is no need to calculate the edit distance, which improves the efficiency of the edit distance calculation, thereby improving the efficiency of fuzzy search and reducing the return of erroneous results.

[0137] In this embodiment of the application, the optimization method may further include:

[0138] Obtain synonyms and specialized terms in the target field;

[0139] A knowledge base is built based on synonyms and specialized terms in the target field.

[0140] Specifically, the target domain refers to the technical field required by the user. The knowledge base in this application embodiment may include a library of professional terms and synonyms related to the current technical field. Taking the engineering machinery field as an example, the synonyms in the knowledge base may include, but are not limited to: "concrete outlet": ["concrete outlet gate", "unloading port", "discharge gate", "unloading gate"]; "feed inlet": ["feed inlet", "feed gate"]; "level gauge": ["level gauge"]; "thermal overload device": ["thermal overload relay"], etc. The professional terms in the knowledge base may include, but are not limited to: "sand silo", "vibrating motor", "aggregate scale", "screw conveyor", "main machine door", "intermediate silo", "sand and gravel silo", "intermediate compartment door", "aggregate compartment door", etc. The processor can acquire synonyms and professional terms in the target domain, and then construct a knowledge base based on these synonyms and professional terms. Acquiring synonyms and professional terms in the target domain can be done by receiving information input from relevant personnel, or by accumulating synonyms and professional terms in related fields through machine learning algorithms. The semantic information of strings is enriched by building a knowledge base.

[0141] In this embodiment of the application, the optimization method may further include:

[0142] Update the knowledge base upon receiving new input synonyms and / or specialized terms.

[0143] Specifically, to improve the accuracy of fuzzy search, the knowledge base needs to be updated in real time. Therefore, when the processor receives new synonyms and / or specialized terms, it needs to update the knowledge base to improve the accuracy of fuzzy search.

[0144] Figure 5 A schematic block diagram illustrating an optimization apparatus for fuzzy search according to an embodiment of this application is shown. Figure 5 As shown in the figure, this application embodiment provides an optimization device for fuzzy search, which may include:

[0145] Memory 510 is configured to store instructions; and

[0146] The processor 520 is configured to retrieve instructions from memory 510 and, when executing instructions, to implement the aforementioned optimization method for fuzzy search.

[0147] Specifically, in this embodiment of the application, the processor 520 can be configured to:

[0148] Retrieve the input query string and multiple document strings stored in the knowledge base, respectively;

[0149] Tokenize the query string to obtain multiple query substrings;

[0150] Tokenize any document string from multiple document strings to obtain multiple document substrings from any document string;

[0151] For any document substring, determine whether the query string contains a target query substring that has no fuzzy similarity to the arbitrary document substring;

[0152] If a target query substring exists in the query string and has no fuzzy similarity to any document substring, the target query substring is replaced with an empty string, and the edit distance between the target query substring and any document substring is set to a preset value.

[0153] In the embodiments of this application, multiple query substrings and multiple document substrings of any document string all include word segmentation arrays and part-of-speech arrays, and the part-of-speech arrays and word segmentation arrays correspond one-to-one.

[0154] Furthermore, the processor 520 can also be configured as follows:

[0155] Determining whether a query string contains a target query substring that has no fuzzy similarity to any document substring includes at least one of the following:

[0156] Based on the word segmentation arrays and part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether the query string contains a query substring that matches the characters of any document substring but has a different part of speech; if the query string contains a query substring that matches the characters of any document substring but has a different part of speech, determine whether the query string contains a target query substring that has no fuzzy similarity to any document substring; or

[0157] Obtain the entity sets of the query string and any document string; based on the word segmentation array, part-of-speech array, and entity set, determine whether the query string contains a substring that matches the characters of any document substring, has the same part of speech but a different entity type; if the query string contains a substring that matches the characters of any document substring, has the same part of speech but a different entity type, determine whether the query string contains a target query substring with no fuzzy similarity to any document substring; or

[0158] Obtain the entity sets of the query string and any document string, as well as the thesaurus of the knowledge base; based on the word segmentation array, part-of-speech array, entity set, and thesaurus, determine whether the query string contains a substring that matches the character set, part of speech, and entity type of any document substring but is not a synonym; if the query string contains a substring that matches the character set, part of speech, and entity type of any document substring but is not a synonym, determine whether the query string contains a target substring with no fuzzy similarity to any document substring.

[0159] Furthermore, the processor 520 can also be configured as follows:

[0160] Based on the word segmentation arrays and part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether there exists a query substring in the query string that matches characters of any document substring but has a different part of speech, including:

[0161] Based on the tokenized arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are characters in the query string that intersect with any document substring;

[0162] If the query string contains characters that intersect with any document substring, retrieve the similar query substring and the similar document substring.

[0163] Based on the part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether similar query substrings and similar document substrings have the same part of speech;

[0164] When similar query substrings and similar document substrings have different parts of speech, it is determined that the query string contains a query substring that matches the characters of any document substring but has a different part of speech.

[0165] Furthermore, the processor 520 can also be configured as follows:

[0166] Determine whether a query string exists that matches any document substring in terms of characters, part-of-speech tagging, but different entity type, based on the word segmentation array, part-of-speech tagging array, and entity set. This includes:

[0167] Based on the tokenized arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are characters in the query string that intersect with any document substring;

[0168] If the query string contains characters that intersect with any document substring, retrieve the similar query substring and the similar document substring.

[0169] Based on the part-of-speech arrays of multiple query substrings and multiple document substrings of any document string, determine whether similar query substrings and similar document substrings have the same part of speech;

[0170] If the parts of speech of similar query substrings and similar document substrings are the same, determine whether the entity types of similar query substrings and similar document substrings are the same based on the entity set;

[0171] When the entity types of similar query substrings and similar document substrings are different, it is determined that the query string contains a query substring that matches any document substring in terms of characters, has the same part of speech, but has a different entity type.

[0172] Furthermore, the processor 520 can also be configured as follows:

[0173] Based on the word segmentation array, part-of-speech array, entity set, and the thesaurus, determine whether a query string contains a substring that matches any document substring in terms of character type, part of speech, and entity type, but is not a synonym.

[0174] Based on the tokenized arrays of multiple query substrings and multiple document substrings of any document string, determine whether there are characters in the query string that intersect with any document substring;

[0175] If the query string contains characters that intersect with any document substring, retrieve the similar query substring and the similar document substring.

[0176] Based on the part-of-speech arrays of multiple query substrings and multiple document substrings of any document substring, determine whether similar query substrings and similar document substrings have the same part of speech;

[0177] If the parts of speech of similar query substrings and similar document substrings are the same, determine whether the entity types of similar query substrings and similar document substrings are the same based on the entity set;

[0178] If the entity types of similar query substrings and similar document substrings are the same, determine whether the similar query substrings and similar document substrings are synonyms based on the thesaurus;

[0179] If the similar query substring and the similar document substring are not synonyms, it is determined that the query string contains a query substring that matches any document substring in terms of characters, part of speech, and entity type, but is not a synonym.

[0180] Furthermore, the processor 520 can also be configured as follows:

[0181] Obtain synonyms and specialized terms in the target field;

[0182] A knowledge base is built based on synonyms and specialized terms in the target field.

[0183] Furthermore, the processor 520 can also be configured as follows:

[0184] Update the knowledge base upon receiving new input synonyms and / or specialized terms.

[0185] The above technical solution involves obtaining the input query string and multiple document strings stored in the knowledge base. The query string and any document strings are then segmented to obtain a query substring and multiple document substrings of the arbitrary document string. For any document substring, if the query string contains a target query substring with no fuzzy similarity to the arbitrary document substring, the target query substring is replaced with an empty string, and the edit distance between the target query substring and the arbitrary document substring is set to a preset value. By replacing query substrings with no fuzzy similarity with empty strings, the number of incorrect results returned by fuzzy search is reduced. Furthermore, setting the edit distance between the target query substring with no fuzzy similarity and the arbitrary document substring to a preset value reduces the calculation of edit distance, thereby improving the efficiency of fuzzy search.

[0186] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described optimization method for fuzzy search.

[0187] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0188] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0189] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0190] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0191] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0192] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0193] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0194] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0195] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An optimization method for fuzzy search, characterized in that, The optimization method includes: Retrieve the input query string and multiple document strings stored in the knowledge base, respectively; The query string is segmented into multiple query substrings; Tokenize any document string from the plurality of document strings to obtain multiple document substrings from the arbitrary document string; For any document substring, determine whether the query string contains a target query substring that has no fuzzy similarity to the arbitrary document substring; If the query string has a target query substring that has no fuzzy similarity to the arbitrary document substring, the target query substring is replaced with an empty string, and the edit distance between the target query substring and the arbitrary document substring is determined to a preset value. The plurality of query substrings and the plurality of document substrings of the arbitrary document string each include a word segmentation array and a part-of-speech array, wherein the part-of-speech array and the word segmentation array correspond one-to-one; The determination of whether the query string has a target query substring with no fuzzy similarity to the arbitrary document substring includes at least one of the following: Based on the word segmentation arrays and part-of-speech arrays of the multiple query substrings and the multiple document substrings of the arbitrary document string, determine whether the query string contains a query substring that matches the characters of the arbitrary document substring but has a different part of speech; if the query string contains a query substring that matches the characters of the arbitrary document substring but has a different part of speech, determine whether the query string contains a target query substring that has no fuzzy similarity to the arbitrary document substring; or Obtain the entity set of the query string and the arbitrary document string; determine whether the query string has a substring that matches the characters of the query string, has the same part of speech but a different entity type, based on the word segmentation array, the part-of-speech array, and the entity set; if the query string has a substring that matches the characters of the arbitrary document string, has the same part of speech but a different entity type, determine whether the query string has a target query substring with no fuzzy similarity to the arbitrary document substring; or Obtain the entity set of the query string and the arbitrary document string, as well as the thesaurus of the knowledge base; determine whether the query string exists in a substring that matches the arbitrary document substring in terms of character matching, part-of-speech, and entity type but is not a synonym, based on the word segmentation array, the part-of-speech array, the entity set, and the thesaurus; if the query string exists in a substring that matches the arbitrary document substring in terms of character matching, part-of-speech, and entity type but is not a synonym, determine that the query string exists in a target query substring that has no fuzzy similarity to the arbitrary document substring.

2. The optimization method according to claim 1, characterized in that, The step of determining whether there exists a query substring in the query string that matches the characters of the arbitrary document substring but has a different part of speech based on the word segmentation array and part-of-speech array of the multiple query substrings and the multiple document substrings of the arbitrary document string includes: Based on the word segmentation arrays of the multiple query substrings and the multiple document substrings of the arbitrary document string, determine whether there are any characters in the query string that intersect with the arbitrary document substring; If the query string contains characters that intersect with any document substring, obtain the similar query substring and the similar document substring. Based on the part-of-speech arrays of the multiple query substrings and the multiple document substrings of the arbitrary document string, determine whether the similar query substrings and the similar document substrings have the same part of speech; If the parts of speech of the similar query substring and the similar document substring are different, it is determined that there exists a query substring in the query string that matches the characters of any document substring but has a different part of speech.

3. The optimization method according to claim 1, characterized in that, The step of determining whether the query string exists in the context of the word segmentation array, the part-of-speech array, and the entity set, and whether it has a character match, the same part-of-speech, but a different entity type with any document substring, includes: Based on the word segmentation arrays of the multiple query substrings and the multiple document substrings of the arbitrary document string, determine whether there are any characters in the query string that intersect with the arbitrary document substring; If the query string contains characters that intersect with any document substring, obtain the similar query substring and the similar document substring. Based on the part-of-speech arrays of the multiple query substrings and the multiple document substrings of the arbitrary document string, determine whether the similar query substrings and the similar document substrings have the same part of speech; If the similar query substring and the similar document substring have the same part of speech, determine whether the entity types of the similar query substring and the similar document substring are the same based on the entity set; If the entity types of the similar query substring and the similar document substring are different, it is determined that the query string has a query substring that matches the characters of any document substring, has the same part of speech but a different entity type.

4. The optimization method according to claim 1, characterized in that, The step of determining whether a query string exists that matches any document substring in terms of character type, part-of-speech tag, entity type, and the thesaurus, based on the word segmentation array, the part-of-speech tag array, the entity set, and the thesaurus, but is not a synonym, includes: Based on the word segmentation arrays of the multiple query substrings and the multiple document substrings of the arbitrary document string, determine whether there are any characters in the query string that intersect with the arbitrary document substring; If the query string contains characters that intersect with any document substring, obtain the similar query substring and the similar document substring. Based on the part-of-speech arrays of the multiple query substrings and the multiple document substrings of the arbitrary document substring, determine whether the similar query substring and the similar document substring have the same part of speech; If the similar query substring and the similar document substring have the same part of speech, determine whether the entity types of the similar query substring and the similar document substring are the same based on the entity set; If the entity types of the similar query substring and the similar document substring are the same, determine whether the similar query substring and the similar document substring are synonyms based on the thesaurus. If the similar query substring and the similar document substring are not synonyms, it is determined that the query string has a query substring that matches the characters of any document substring, has the same part of speech, and has the same entity type but is not a synonym.

5. The optimization method according to claim 1, characterized in that, The optimization method further includes: Obtain synonyms and specialized terms in the target field; A knowledge base is constructed based on synonyms and specialized terms in the target field.

6. The optimization method according to claim 5, characterized in that, The optimization method further includes: The knowledge base is updated upon receiving new input synonyms and / or specialized terms.

7. An optimization device for fuzzy search, characterized in that, include: The memory is configured to store instructions; as well as A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the optimization method for fuzzy search according to any one of claims 1 to 6.

8. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the optimization method for fuzzy search according to any one of claims 1 to 6.

Images