Character string retrieval method and device, computer equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a character string retrieval method and device and a computer readable storage medium. The method comprises the following steps: acquiring a given reference character string set to perform index structure construction operation; according to the obtained retrieval instruction, k-nearest neighbor retrieval operation is executed for the index structure, and a k-nearest neighbor approximate retrieval result is returned. The index structure is composed of a neighbor graph and a corresponding table; the neighbor graph is composed of directed graphs, each layer of directed graph is composed of nodes corresponding to a reference character string, part of the nodes of each layer are defined as center points, and the nodes of the layer are divided into clusters with limited sizes; the corresponding table stores information of each center point, a cluster where the center point is located and a nearest neighbor center point set. In the retrieval process realized on the basis of the constructed index structure, the number of times of the edit distance needing to be calculated is far smaller than that of the existing method. Therefore, according to the character string k-nearest neighbor likelihood retrieval algorithm aiming at the editing distance, the time complexity is obviously lower than that in the prior art, and character string retrieval can be more efficiently achieved.

Description

Technical Field

[0001] The present application belongs to the field of natural language processing, and in particular, relates to a string retrieval method, apparatus, computer device and computer-readable storage medium. Background Art

[0002] Edit distance is a common measure of literal proximity between two strings. It is defined as the sum of the cost of converting one string to another by adding, deleting, and modifying characters. Adding, deleting, and modifying characters are three common operations, each with a preset unit cost. The string k-nearest neighbor retrieval problem based on edit distance is a typical problem in the field of natural language processing. It is generally defined as finding k reference strings with the smallest edit distance to the query string from the reference string set under a given unit cost of adding, deleting, and modifying characters, a given reference string set, and a given query string, and listing these k reference strings in order of edit distance from small to large. This problem has a wide range of applications in scenarios such as text error correction, automatic completion, and speech recognition.

[0003] Although the calculation of the k-nearest neighbors of a string based on edit distance has a wide range of applications, the problem has a high computational complexity, which limits the practicality of the solution to this problem. On the one hand, the optimal dynamic programming method is used to calculate the edit distance between two strings of length m and n, and the time complexity reaches O(mn), which is the square of the string length. On the other hand, to find the k nearest neighbors of a given query string among M reference strings, it is usually necessary to first calculate M edit distances and then select the smallest k from these M edit distances. Since the dynamic programming algorithm of the edit distance is difficult to further optimize, in order to reduce the computational complexity of the k-nearest neighbors of a string based on edit distance, it is necessary to introduce some index structures to reduce the number of edit distance calculations. In the field of k-nearest neighbor retrieval of vectors, there is a class of efficient approximate k-nearest neighbor retrieval algorithms. The basic idea of ​​this type of algorithm is to introduce a specific index structure to ensure that the query vector can find the approximate k-nearest neighbors under the premise of calculating the proximity between some but not all reference vectors and the query vector. However, existing methods for calculating the k-nearest neighbors of a string based on edit distance (including adapted methods for approximate k-nearest neighbor calculation in the field of vector retrieval) have efficiency or effectiveness issues, and more efficient methods need to be designed to meet the requirements of fast response and high accuracy in practical applications. How to improve the efficiency of string retrieval is a technical problem that needs to be solved urgently by those skilled in the art.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0005] Based on this, it is necessary to propose a string retrieval method, apparatus, computer device and computer-readable storage medium to address the above problems, which can achieve string retrieval efficiency more efficiently.

[0006] The present application solves the technical problem by adopting the following technical solutions: The present application provides a string retrieval method, comprising the following steps: when there is no index structure, obtaining a given reference string set to perform an index structure construction operation; the index structure is composed of a neighbor graph and a correspondence table; the neighbor graph is composed of Layer directed graph, is a preset integer; each layer of the directed graph is composed of nodes, and one node corresponds to a reference string; The layer directed graph stores all given reference strings; for no less than 1 and no greater than Any integer , No. The reference string set stored in the layer directed graph is A subset of the reference string set stored in a layer directed graph; some nodes of each layer directed graph are marked as center points, which are determined by the index structure construction operation. The center points divide all nodes in the directed graph into multiple non-overlapping subsets, each subset is called a cluster, and the number of nodes in the cluster does not exceed , is a preset integer; in each layer of the directed graph, there are directed edges between some nodes, and the directed edges are determined according to the index structure construction operation; the nodes pointed by the directed edges are called neighbors of the node initiating the directed edge, and the number of neighbors of each node does not exceed ; Each layer of the neighbor graph has a unique corresponding table, which is a two-dimensional table composed of rows and columns; each row corresponds to a center point of the corresponding layer in the neighbor graph; there are three columns in total: the first column stores the reference string corresponding to the center point; the second column stores the reference string corresponding to other nodes in the cluster where the center point is located except the center point; the third column stores the reference string set corresponding to the nearest neighbor center point set of the center point, and the nearest neighbor center point set of the center point is determined by the index structure construction operation; when the reference string set corresponding to the index structure changes, the add, delete and modify instructions are generated according to the changes in the reference string set, and the add, delete and modify operations are performed on the index structure according to the add, delete and modify instructions; according to the obtained retrieval instructions, the index structure is executed -Nearest neighbor search operation, return - The search results output by the nearest neighbor search operation; the search instruction content includes the query string and the expected result set size and retrieval distance, the retrieval distance is the preset edit distance from one string to another string; the retrieval operation includes: marking the query string as a test node; randomly selecting Nodes without deletion marks are marked as the entry nodes of the first layer. The deletion marks of the nodes are determined according to the index structure. If the number of nodes without deletion marks in the first layer directed graph is less than , all nodes without deletion marks are marked as the entry nodes of the first-layer directed graph; the processing operation is performed layer by layer starting from the first layer of the neighbor graph until the first layer of the neighbor graph. The layer processing is completed, and the processing operation includes: After the entry node set of the layer directed graph is obtained, the entry node set is first marked as the node set at time 0 ; Then according to the gradually increasing time , in turn by Time Node Set Determine Time Node Set ,Will The union of the neighbor sets of all nodes in is marked as the first union, and the first union is combined with The union of is marked as the second union, then Determine the node with the smallest retrieval distance to the test node in the second union The set is composed of nodes without deletion marks; if the number of nodes without deletion marks in the second union is less than , then Determine the set consisting of all nodes without deletion marks in the second union; when = Time will Mark as ;like , then Marked as The entry node set of the layer directed graph and continue Layer processing; if , then Mark as search result and output.

[0007] In an optional embodiment of the present application, the index structure construction operation includes: determining that the minimum level at which each reference string is added to the neighbor graph is the first l Probability of layer : ,in is a preset real number greater than 1; according to Randomly determine the minimum level l at which each given reference string is added to the neighbor graph, and add the given reference strings as nodes to the first level of the neighbor graph one by one. Layer to In the directed graph of each layer, when adding a corresponding node of a reference string in the directed graph of each layer, the node to be added is marked as an insertion node; the first insertion node is regarded as the first center point and forms a cluster by itself; in the subsequent insertion process, the specified distance value between the newly inserted node and the existing center point is calculated one by one, and the specified distance value is a preset literal distance measurement value between the two nodes that satisfies non-negativity, symmetry and triangle inequality; the cluster where the center point with the smallest specified distance value is located is marked as an insertion cluster, and the insertion node is added to the insertion cluster; when the number of nodes in the insertion cluster exceeds , split the inserted cluster into two new clusters, the center points of the two new clusters are two different nodes in the inserted cluster, satisfying that the sum of the deviation values ​​from other nodes in the inserted cluster to the two new clusters is the minimum, and the deviation value of each node to the two new clusters is the minimum value of the specified distance value between the node and the center points of the two new clusters. Then, the other nodes in the extended cluster except the center points of the two new clusters are called the nodes to be processed, and the specified distance values ​​between the nodes to be processed and the center points of the two new clusters are calculated, and the nodes to be processed are added to the new cluster with the smaller specified distance value; if the specified distance values ​​between the nodes to be processed and the center points of the two new clusters are the same, the nodes to be processed are added to the new cluster with fewer nodes; after all given reference strings are added as nodes to the neighbor graph, determine the clusters included in each layer of the directed graph of the neighbor graph; calculate the node with the smallest sum of the specified distance values ​​from other nodes in each cluster, and mark it as the center point of each cluster; for those nodes that satisfy Every integer , according to the neighbor graph All the center points and other nodes in the layer determine the neighbor relationship to build the first The corresponding table of each layer is summarized into the neighbor graph and its corresponding table of each layer to obtain the index structure.

[0008] In an optional embodiment of the present application, according to the first All the center points and other nodes in the layer determine the neighbor relationship to build the first The corresponding table of the layer includes: The specified distance value between the center points in the layer; for each center point, the specified distance value between the center points is determined other center points with the smallest specified distance value. The set of other center points is marked as the nearest neighbor center point set of the center point; l For each node in the layer, obtain the center point of the cluster where the node is located, and all the center points in the set of nearest neighbor center points of the center point of the cluster, a total of center point; The set of center points is marked as the node's nearest neighbor center point set; get the cluster of all center points in the nearest neighbor center point set of each node, with the smallest distance value to the node Other nodes will The set of other nodes is marked as the neighbor set of the node; Add nodes to the layer directed graph directed edges of other nodes; according to The reference string corresponding to each center point in the layer, the reference string corresponding to other nodes in the cluster where the center point is located, and the set of nearest neighboring center points of the center point are used to determine the first Correspondence table of layers.

[0009] In an optional embodiment of the present application, the add, delete and modify instruction includes an operation type and an operation object, the operation type is one of add, delete and modify; when the operation type is add, the operation object is a reference string to be added; when the operation type is delete, the operation object is a reference string to be deleted; when the operation type is modify, the operation object is a reference string before the change and a reference string after the change.

[0010] In an optional embodiment of the present application, the index structure is added, deleted, and modified according to the addition, deletion, and modification instructions, including: when the operation type of the addition, deletion, and modification instructions is to add, obtain the reference string to be added corresponding to the operation object, and mark the reference string to be added as an insertion node; determine that the minimum layer of the neighbor graph to be inserted is the first Probability of layer : , is a preset real number greater than 1; according to Randomly determine the minimum level at which the inserted node will be added , and add the inserted node to the neighbor graph Layer to When adding an inserted node to each directed graph, the specified distance between the inserted node and all the center points in the directed graph is calculated, the cluster where the center point with the smallest specified distance value is located is marked as an extended cluster, and the inserted node is added to the extended cluster; When the number of nodes in the expanded cluster after adding the inserted node exceeds When , the extended cluster is split into two new clusters, and the center points of the two new clusters are two different nodes in the extended cluster, so that the sum of the deviation values ​​of other nodes in the extended cluster to the two new clusters is minimized, and the deviation value of each node to the two new clusters is the minimum value of the specified distance value between the node and the center points of the two new clusters; the other nodes in the extended cluster except the center points of the two new clusters are called the nodes to be processed, and the specified distance values ​​between the nodes to be processed and the center points of the two new clusters are calculated; the nodes to be processed are added to the new cluster with the smaller specified distance value; if the specified distance values ​​between the nodes to be processed and the center points of the two new clusters are the same, the nodes to be processed are added to the new cluster with fewer nodes; the specified distance values ​​between the center points of the two new clusters and other center points are calculated, and the center point of each new cluster with the smallest specified distance value is determined according to the specified distance value. Other center points will The set of other center points is marked as the nearest neighbor center point set of the center point of the new cluster; for each node in the two new clusters, obtain the center point of each node corresponding to the new cluster and all the center points in the nearest neighbor center point set of the center point of the new cluster, a total of center point; The set of centers is marked as the nearest neighbor center set of each node in the new cluster; get the cluster where all the centers in the nearest neighbor center set of each node are located, and the one with the smallest distance value to the node Other nodes will The set of other nodes is marked as the neighbor set of the node; the row where the center point of the extended cluster is located is deleted from the corresponding table, and the corresponding row information of the center points of the two new clusters is added to the corresponding table according to the node sets, center points and the nearest neighbor center point sets of the center points of the new clusters; each node in the neighbor set of the inserted node that does not belong to any new cluster is marked as a neighbor to be processed, the inserted node is first added to the neighbor set of the neighbor to be processed, and then the neighbor with the largest specified distance value is deleted from the neighbor set of the neighbor to be processed; When the number of nodes in the expanded cluster after adding the inserted node does not exceed When , get the center point of the cluster where the inserted node is located, and all the center points in the set of nearest neighbor center points of the center point of the cluster, a total of center point; The set of center points is marked as the nearest neighbor center point set of the inserted node; get the cluster of all center points in the nearest neighbor center point set of the inserted node, and the one with the smallest specified distance value to the inserted node other nodes; The set of other nodes is marked as the neighbor set of the inserted node; the second column information of the row where the center point of the extended cluster is located in the corresponding table is updated according to the node set of the extended cluster; each node in the neighbor set of the inserted node is marked as a neighbor to be processed; the inserted node is first added to the neighbor set of the neighbor to be processed, and then the neighbor with the largest specified distance value is deleted from the neighbor set of the neighbor to be processed.

[0011] In an optional embodiment of the present application, the addition, deletion and modification operations are performed on the index structure according to the addition, deletion and modification instructions, including: when the operation type of the addition, deletion and modification instructions is deletion, obtaining a reference string to be deleted corresponding to the operation object, marking the reference string to be deleted as a deletion node; processing the directed graph and the corresponding table containing the deletion node layer by layer for the neighbor graph; determining the cluster where the deletion node is located according to the corresponding table, marking the cluster where the deletion node is located as a contraction cluster, and marking the deletion node with a deletion mark; When all nodes in the contracted cluster have deletion marks, delete all nodes in the contracted cluster from the directed graph, and delete the row where the center point of the contracted cluster is located from the corresponding table; for each center point in the directed graph, if the center point's nearest neighbor center point set contains the center point of the contracted cluster, then re-determine the node in the directed graph with the smallest specified distance value to the center point. other center points, and update the center point's nearest neighbor center point set to be composed of these For each node in the directed graph, if the node's neighbor set contains at least one node in the contracted cluster, then re-obtain the center point of the cluster where the node is located, and all the center points in the nearest neighbor center point set of the center point of the cluster, for a total of center point; The set of center points is updated to the node's nearest neighbor center point set; the cluster of all center points in the node's nearest neighbor center point set is re-obtained, and the one with the smallest distance value to the node is Other nodes will The set of other nodes is updated as the neighbor set of the node; If there are still nodes without deletion marks in the shrunk cluster, no further processing is performed.

[0012] In an optional embodiment of the present application, add, delete and modify operations are performed on the index structure according to the add, delete and modify instructions, including: when the operation type of the add, delete and modify instructions is modification, obtaining the reference string before change and the reference string after change corresponding to the operation object; first marking the reference string before change as a deletion node to perform the add, delete and modify operations with the operation type of deletion, and then marking the reference string after change as an insertion node to perform the add, delete and modify operations with the operation type of addition.

[0013] The present application also provides a string search device, including: an index construction module, which is used to obtain a given reference string set to perform an index structure construction operation when there is no index structure; an index maintenance module, which is used to generate an add, delete, or modify instruction according to the change of the reference string set when the reference string set corresponding to the index structure changes, and perform add, delete, or modify operations on the index structure according to the add, delete, or modify instructions; a search module, which is used to perform a search on the index structure according to the obtained search instructions. -Nearest neighbor search operation, returns the search result output by the search operation.

[0014] The present application also provides a computer device, including a processor and a memory: the processor is used to execute a computer program stored in the memory to implement the aforementioned method.

[0015] The present application also provides a computer-readable storage medium storing a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0016] The embodiments of the present application have the following beneficial effects: This application is based on The key to the index structure of the layer neighbor graph is to limit the number of neighbors of each node, so that the retrieval process can be performed on the limited neighborhood of the entry node set rather than the entire set of nodes, thereby controlling the number of nodes traversed during the retrieval process. In order to limit the number of neighbors of the inserted node to a preset threshold The index construction method directly adapted to HNSW needs to find the node with the smallest specified distance value to the inserted node from all added nodes. nodes, so the index construction method directly adapted to HNSW requires calculation times the specified distance value, where is the number of reference strings. The most expensive operation in the string retrieval method based on this index structure is the operation of calculating the edit distance between two strings. Therefore, the number of edit distance calculations can be used to measure the time complexity of the string retrieval method. Each layer in the layer neighbor graph needs to calculate Edit distance, where is the number of layers of the neighbor graph, is the number of nodes included in the search results, is the maximum number of nodes in a cluster, is the average number of iterations at each layer convergence, so the above string retrieval method needs to calculate a total of times the edit distance. In contrast, the original string retrieval method needs to calculate edit distance. is small (usually between 1 and 9), and is much smaller than Therefore, the string retrieval method of this application can be regarded as the same as The time complexity of the unrelated constant-level approximate retrieval algorithm is significantly lower than the linear level of the original retrieval algorithm.

[0017] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with the accompanying drawings. It should be understood that the above general description and the detailed description below are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] in: Figure 1 A flowchart of a character string search method provided by an embodiment; Figure 2 A diagram showing the relationship between internal functional modules of a string search device provided by an embodiment; Figure 3 A schematic block diagram of the structure of a computer device provided by an embodiment. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] Edit distance is a common measure of literal proximity between two strings. It is defined as the sum of the cost of converting one string to another by adding, deleting, and modifying characters. Adding, deleting, and modifying characters are three common operations, each with its own preset unit cost. When the unit cost of adding and deleting characters is different, the edit distance is asymmetric, that is, the edit distance from string A to string B is not necessarily equal to the edit distance from B to A. The string k-nearest neighbor retrieval problem based on edit distance is a typical problem in the field of natural language processing. It is generally defined as finding k reference strings with the smallest edit distance to the query string from the reference string set under a given unit cost of adding, deleting, and modifying characters, a given reference string set, and a given query string, and listing these k reference strings in order of edit distance from small to large. This problem has a wide range of applications in scenarios such as text error correction, automatic completion, and speech recognition. Among them, in the text correction scenario, calculating the k-nearest neighbors of a string based on the edit distance can be used to detect spelling errors and correct grammatical errors; in the automatic completion scenario, calculating the k-nearest neighbors of a string based on the edit distance can be used to automatically complete the words entered by the user, that is, automatically completing the words by finding the words corresponding to the minimum edit distance; in the speech recognition scenario, calculating the k-nearest neighbors of a string based on the edit distance can improve the accuracy of speech recognition by identifying words in the speech and finding the most likely text match.

[0022] Although computing the k-nearest neighbors of a string based on edit distance has a wide range of applications, the problem has a high computational complexity, which limits the practicality of the solution to this problem. On the one hand, the time complexity of computing the edit distance between two strings of length m and n using the optimal dynamic programming method is O(mn), which is the square of the string length. On the other hand, to find the k nearest neighbors of a given query string among M reference strings, it is usually necessary to first calculate M edit distances and then select the smallest k from these M edit distances. Since the dynamic programming algorithm of the edit distance is difficult to further optimize, in order to reduce the computational complexity of the k-nearest neighbors of a string based on edit distance, it is necessary to introduce some index structures to reduce the number of edit distance calculations.

[0023] In the field of vector k-nearest neighbor retrieval, there is a class of efficient approximate k-nearest neighbor retrieval algorithms. The basic idea of ​​this type of algorithm is to introduce a specific index structure to ensure that the query vector can find the approximate k-nearest neighbors under the premise of calculating the proximity between some but not all reference vectors and the query vector. For example, the popular Hierarchical Navigable Small World (HNSW) algorithm constructs a multi-layer neighbor graph as an index structure, where the nodes of the neighbor graph are reference vectors, and the number of neighbors of each layer of nodes is limited to a preset threshold; when calculating the approximate k-nearest neighbors of a given query vector, the HNSW algorithm starts from the sparsest top-level neighbor graph and refines the approximate k-nearest neighbors of the query vector layer by layer, that is, using the approximate k-nearest neighbors output by the adjacent upper layer as the entry, and looking for more accurate approximate k-nearest neighbors in the current layer until the bottom-level neighbor graph containing all reference vectors is processed. This algorithm only needs to traverse a small number of vector nodes in each layer of the multi-layer neighbor graph and calculate the proximity between them and the query vector, so it does not need to calculate the proximity between all reference vectors and the query vector.

[0024] There are two relatively intuitive methods to adapt the approximate k-nearest neighbor retrieval algorithm in the field of vector retrieval to the calculation of string k-nearest neighbors based on edit distance.

[0025] One method is to vectorize the string first, then build a vector-based index structure, and finally call a similar approximate k-nearest neighbor retrieval algorithm to find the approximate k-nearest neighbors based on edit distance. For example, algorithms such as HNSW require that two connected vector nodes in the neighbor graph be as close as possible in the specified vector proximity, but two vector nodes that are as close as possible in vector proximity are not necessarily close in edit distance, because the edit distance considers literal symbols, and the value depends on the preset unit cost of the three operations of addition, deletion and modification, while the vector proximity considers the semantics of the word. This results in that in a multi-layer neighbor graph constructed using vector nodes, it is not necessarily possible to gradually find a more accurate approximate nearest neighbor based on edit distance through neighbor relationships.

[0026] Another method does not introduce the vector representation of strings, but directly operates on the symbol level of strings during the index construction process and string retrieval process, and replaces the vector proximity with the edit distance. Since the edit distance is asymmetric, the asymmetry of the node proximity must be considered when replacing the vector proximity with the edit distance, which will inevitably increase the complexity of the graph-based index construction and string retrieval. In addition, this method also aggravates the high cost problem of index construction. Take the HNSW algorithm as an example. The algorithm builds the index structure by adding vector nodes one by one to the multi-layer neighbor graph. When adding a single vector node, it is necessary to calculate the proximity between it and the existing vector nodes in the graph, so a total of M(M-1) / 2 vector proximity calculations are required. Since the computational complexity of vector proximity is linear in the vector dimension, and the computational complexity of edit distance is square in the string length, directly using the multi-layer neighbor graph construction algorithm in HNSW to construct an index structure based on edit distance will incur a higher time cost.

[0027] It can be seen that the existing methods for calculating the k-nearest neighbors of a string based on edit distance (including the adaptation method of approximate k-nearest neighbor calculation in the field of vector retrieval) have efficiency or effectiveness problems, high computational time complexity, and low efficiency. To this end, this application proposes a string retrieval method to meet the requirements of fast response and high accuracy of string retrieval in practical applications. To clearly describe the method provided in this embodiment, please refer to Figure 1 , including steps S110~S130.

[0028] Step S110: when no index structure exists, obtaining a given reference string set to perform an index structure construction operation.

[0029] In one embodiment, the retrieval method provided by the present application reduces the computational time complexity in the retrieval process by introducing an index structure. The index structure is composed of a neighbor graph and a correspondence table. If it is detected that the index structure does not exist before the retrieval, it is necessary to obtain a given reference string set to perform an index structure construction operation to complete the construction of the index structure. The specific composition of the index structure and the construction process will be described in detail later.

[0030] The neighbor graph in the index structure is given by Layer directed graph, is a preset integer. Each layer of the directed graph is composed of nodes, and one node corresponds to a reference string. The layer directed graph stores all given reference strings; for not less than 1 and not greater than Any integer , No. The reference string set stored in the layer directed graph is A subset of the reference string set stored in the layer directed graph. Specifically, The reference string set stored in the layer directed graph and the The reference string sets stored in the layer directed graph. The ratio of these two sets is random and depends on the minimum layer probability of joining. The minimum layer probability of joining will be calculated and determined for each reference string in the index structure construction operation, which will be described in detail later. Some nodes of each layer of the directed graph are marked as center points, which are determined according to the index structure construction operation. The center points divide all nodes in the directed graph into multiple non-overlapping subsets, each subset is called a cluster, and the number of nodes in a cluster does not exceed , is a preset integer. In each layer of the directed graph, there are directed edges between some nodes, which are determined by the index structure construction operation; the nodes pointed by the directed edges are called neighbors of the node initiating the directed edge, and the number of neighbors of each node does not exceed , That is, the preset integer described above will be mentioned many times in the following text and will not be explained separately. The existence of two nodes with directed edges indicates that the reference strings corresponding to the two nodes are relatively close in literal sense.

[0031] Each layer of the neighbor graph has a unique correspondence table, which is a two-dimensional table consisting of rows and columns. Each row corresponds to a center point of the corresponding layer in the neighbor graph; there are three columns in total: the first column stores the reference string corresponding to the center point; the second column stores the reference string corresponding to the nodes in the cluster where the center point is located except the center point; the third column stores the reference string set corresponding to the center point's nearest neighbor center point set, and the center point's nearest neighbor center point set is determined by the index structure construction operation. The nearest neighbor center point set is used to represent the several center points that are closest to a node / center point. There are Center points; the set of nearest neighbor centers of each center point has A center point.

[0032] In one embodiment, the index structure construction operation is performed on a given reference string. As mentioned above, the index structure is composed of a neighbor graph and a correspondence table, and the corresponding construction operation is to respectively construct the neighbor graph and the correspondence table.

[0033] The method provided by this application is implemented through an index structure, which is necessary. Therefore, if it is determined before the search that the index structure required by this application does not exist, it is necessary to construct the index structure accordingly. Assume that the reference string set used to construct the index structure includes When constructing a neighbor graph, you need to process the reference strings one by one to construct the neighbor graph layer by layer. First, determine the minimum layer for each reference string to be added to the neighbor graph.l Probability of layer , The calculation method is as follows: (1) In the above formula, A preset real number greater than 1.

[0034] according to Randomly determine the minimum level at which each given reference string is added to the neighbor graph l , and add the given reference string as a node to the neighbor graph one by one Layer to In the directed graph of the layer, The layer directed graph stores all given reference strings, The reference string set stored in the layer directed graph is A subset of the set of reference strings stored in a layer directed graph. The definition of reference string is added to The total probability of the layer is approximately: (2) The total probability of joining the l+1th layer is approximately: (3) So that the Layer and The ratio of the number of nodes in a layer is approximately equal to: (4) The present application adopts an incremental k-center point clustering algorithm to divide the node set of each layer into non-overlapping clusters, that is, when adding the corresponding node of a reference string in each layer of the directed graph, the node to be added is marked as an insertion node. The first inserted node is regarded as the first center point and forms a cluster by itself. In the subsequent insertion process, the specified distance value of the newly inserted node and the existing center point is calculated one by one. The cluster where the center point with the smallest specified distance value is located is marked as an insertion cluster, and the insertion node is added to the insertion cluster. The specified distance value is a preset literal distance measurement value between two nodes that satisfies non-negativity, symmetry and triangle inequality. Among them, the specified distance value of the two strings can be the difference between the maximum length of the two strings and the maximum common substring length of the two strings, or it can be the edit distance with all operation unit costs set to 1. The requirement that the specified distance value satisfies symmetry is to reduce the calculation amount of the specified distance value by half, and the requirement that the specified distance value satisfies non-negativity and triangle inequality is to ensure that the specified distance value of each node to its neighbor's neighbor is as close as possible to the specified distance value of the node to its neighbor, so as to ensure that there is a greater possibility of finding the nearest neighbor in the neighbor's neighborhood.

[0035] When the number of nodes inserted into the cluster exceeds When , the insertion cluster is split into two new clusters. The centers of the two new clusters are two different nodes in the insertion cluster. The sum of the deviations from other nodes in the insertion cluster to the two new clusters is minimized. The deviation from each node to the two new clusters is the minimum of the specified distances between the node and the centers of the two new clusters. Specifically, the original insertion cluster is marked as , the center points of the two new clusters are selected as and , except for the original inserted cluster and The nodes are marked as . and The following formula needs to be minimized: (5) In the above formula, It is a function for calculating the specified distance value. It should be noted that the specified distance value introduced in the technical solution of the present application during index construction and subsequent updates does not limit the specific calculation formula. As long as the specified distance value is defined based on the string symbol level and satisfies non-negativity, symmetry and triangular inequality, no matter what specific calculation formula is used, it should not be considered to exceed the technical scope protected by the present application.

[0036] The nodes in the expanded cluster other than the centers of the two new clusters are called the nodes to be processed, and the specified distance between the nodes to be processed and the centers of the two new clusters is calculated. The nodes to be processed are added to the new cluster with the smaller specified distance. In other words, for the inserted cluster Each non or Node ,like , then Add to The cluster where ; if , then Add to The cluster where .

[0037] If the distance between the node to be processed and the center points of the two new clusters is the same, the node to be processed is added to the new cluster with fewer nodes. , then Add to The cluster and The cluster with fewer elements among the two clusters.

[0038] After all given reference strings are added as nodes to the neighbor graph, determine the clusters included in each layer of the directed graph of the neighbor graph. Calculate the node with the smallest sum of specified distance values ​​from other nodes in each cluster and re-mark it as the center point of each cluster. In other words, the new center point in the cluster where the original center point is located The following formula needs to be minimized: (6) For satisfaction Every integer , according to the neighbor graph All the center points and other nodes in the layer determine the neighbor relationship to build the first Correspondence table of layers.

[0039] In one embodiment, according to the neighbor graph All the center points and other nodes in the layer determine the neighbor relationship to build the first The corresponding table of the layer needs to determine the set of nearest neighbor center points of each node and center point.

[0040] The processing step first determines the set of nearest neighbor center points for each center point, specifically calculating the The specified distance value between the center points in the layer is determined for each center point according to the specified distance value between the center points. other center points with the smallest specified distance value. The set of other center points is marked as the nearest neighbor center point set of the center point.

[0041] Then determine the set of nearest neighbor centers of each node. l For each node in the layer, obtain the center point of the cluster where the node is located, and all the center points in the set of nearest neighbor center points of the center point of the cluster, a total of center point. The set of center points is marked as the node's nearest neighbor center point set.

[0042] Then determine the neighbor set of each node. Get the cluster of all the centers in the nearest neighbor center set of each node, with the smallest distance value to the node. Other nodes will The set of other nodes is marked as the node's neighbor set. Based on the neighbor set, the directed edges between nodes can be determined, that is, Add nodes to the layer directed graph directed edges to other nodes.

[0043] According to the nearest neighbor center point set and neighbor set determined in the previous article, the first For example, The reference string corresponding to each center point in the layer, the reference string corresponding to other nodes in the cluster where the center point is located, and the set of nearest neighboring center points of the center point are used to determine the first The corresponding table of each layer of the neighbor graph is determined.

[0044] Finally, the neighbor graph and its corresponding table at each layer are summarized to obtain the index structure.

[0045] The most expensive operation in the above index building method is to calculate the specified distance value between two strings. Therefore, the time complexity of the index building method can be measured by the number of times the specified distance value is calculated. When adding a reference string, you do not need to calculate the specified distance value; however, you need to calculate the specified distance value from the beginning of adding the first reference string. When adding a reference string, that is, inserting a node, locating the insertion cluster requires The specified distance between the inserted node and the existing center point is calculated. The insertion cluster split requires Calculate the specified distance value once, so adding all reference strings to the corresponding cluster requires Then, adjust the center points of all clusters to The total number of times the specified distance value is calculated to determine the nearest neighbor center point set for each center point is required Finally, the nearest neighbor center point set of each node is in the cluster where the center point is located to determine the total number of neighbors required for each neighbor set. Therefore, the above index construction method is L Each layer of the layer neighbor graph requires a total of times to calculate the specified distance value, so the overall method requires Calculate the specified distance value. Far less than Therefore, the above time complexity is significantly smaller than that of the index construction method directly adapted to HNSW. Calculates the specified distance value.

[0046] Step S120: when the reference string set corresponding to the index structure changes, an add, delete, or modify instruction is generated according to the change of the reference string set; and an add, delete, or modify operation is performed on the index structure according to the add, delete, or modify instruction.

[0047] In one embodiment, it is understood that the index structure is used to record reference strings, and when the reference string changes, the index structure needs to be changed. Therefore, it is necessary to generate add, delete, and modify instructions according to the changes in the reference string to perform add, delete, and modify operations on the index structure to continuously maintain the index structure. Among them, the add, delete, and modify instructions include an operation type and an operation object, and the operation type is one of add, delete, and modify. Different operation types have different corresponding operation objects: when the operation type is add, the operation object is the reference string to be added; when the operation type is delete, the operation object is the reference string to be deleted; when the operation type is modify, the operation object is the reference string before the change and the reference string after the change. The following text will describe how to perform add, delete, and modify operations on the index structure one by one.

[0048] In one embodiment, the index structure is added, deleted, or modified according to the addition, deletion, or modification instruction, including: when the operation type of the addition, deletion, or modification instruction is addition, obtaining the reference string to be added corresponding to the operation object, and marking the reference string to be added as an insertion node. The subsequent processing is similar to the index structure construction operation, which is to determine the minimum layer to be added to the insertion node, from the minimum layer to the minimum layer. Insert nodes are added layer by layer in the directed graph of the layer, and the corresponding table is updated according to the situation after insertion.

[0049] First, determine the minimum level at which to insert a node into the neighbor graph. Probability of layer , the calculation method refers to formula (1). Randomly determine the minimum level at which the inserted node will be added , and add the inserted node to the neighbor graph Layer to Layer directed graph.

[0050] Then determine the specific cluster where the inserted node is inserted, and update the subsequent nearest neighbor center point set and neighbor set. Specifically, when adding an inserted node in each layer of the directed graph, calculate the specified distance value between the inserted node and all the center points in the directed graph, mark the cluster where the center point with the smallest specified distance value is located as the extended cluster, and add the inserted node to the extended cluster. The calculation method of the specified distance value has been described above and will not be elaborated in the following text.

[0051] During the insertion process, the number of nodes in a cluster cannot exceed , so it is necessary to determine whether it exceeds , the insertion process is divided into two ways.

[0052] When the number of nodes in the expanded cluster after adding the inserted node exceeds When , the extended cluster is split into two new clusters. The center points of the two new clusters are determined, specifically two different nodes in the extended cluster, so that the sum of the deviation values ​​from other nodes in the extended cluster to the two new clusters is minimized, and the deviation value of each node to the two new clusters is the minimum value of the specified distance value between the node and the center points of the two new clusters.

[0053] The nodes in the extended cluster except the center points of the two new clusters are called the nodes to be processed, and the specified distance values ​​between the nodes to be processed and the center points of the two new clusters are calculated. The nodes to be processed are added to the new cluster with the smaller specified distance value; if the specified distance values ​​between the nodes to be processed and the center points of the two new clusters are the same, the nodes to be processed are added to the new cluster with fewer nodes.

[0054] Then the nearest neighbor center point set of the center point of the new cluster and the nearest neighbor center point set of each node in the new cluster are updated. The specified distance value between the center point of the two new clusters and other center points is calculated, and the center point of each new cluster with the smallest specified distance value is determined according to the specified distance value. other center points, The set of other center points is marked as the nearest neighbor center point set of the center point of the new cluster. For each node in the two new clusters, obtain the center point of each node corresponding to the new cluster and all the center points in the nearest neighbor center point set of the center point of the new cluster, a total of center point. The set of center points is marked as the nearest neighbor center point set of each node in the new cluster.

[0055] Then update the neighbor sets of the nodes in the new cluster, keeping the number of neighbor sets of each node equal to Get the nearest neighbor center point set of each node in the cluster where all the center points are located, and the one with the smallest distance value to the node specified Other nodes will The set of other nodes is marked as the neighbor set of the node. In addition, each node in the neighbor set of the inserted node that does not belong to any new cluster is marked as a neighbor to be processed. The inserted node is first added to the neighbor set of the neighbor to be processed, and then the neighbor with the largest specified distance value is deleted from the neighbor set of the neighbor to be processed.

[0056] According to the above processing, the update of the neighbor graph is completed, and the subsequent update is the update of the correspondence table. The row where the center point of the extended cluster is located is deleted from the correspondence table, and the corresponding row information of the center points of the two new clusters is added to the correspondence table according to the node set, center point and the nearest neighbor center point set of the center point of the new cluster.

[0057] When the number of nodes in the expanded cluster after adding the inserted node does not exceed When , because there is no need for clustering, the nearest neighbor center point set, neighbor set and correspondence table are directly updated.

[0058] Determine the nearest neighbor center point set of the inserted node: Get the center point of the cluster where the inserted node is located, and all the center points in the nearest neighbor center point set of the cluster center point, a total of center point. The set of center points is marked as the nearest neighbor center point set of the inserted node.

[0059] Determine the neighbor set of the inserted node: Get the cluster of all the center points in the nearest neighbor center point set of the inserted node, and the one with the smallest specified distance value to the inserted node other nodes. The set of other nodes is marked as the neighbor set of the inserted node. In addition, each node in the neighbor set of the inserted node is marked as a neighbor to be processed. The inserted node is first added to the neighbor set of the neighbor to be processed, and then the neighbor with the largest specified distance value is deleted from the neighbor set of the neighbor to be processed, so as to maintain the number of elements in the neighbor set of the neighbor to be processed. .

[0060] Update the corresponding table: update the second column information of the row where the center point of the extended cluster is located in the corresponding table according to the node set of the extended cluster.

[0061] The incremental method described above for adding a single reference string is Each layer of the layer neighbor graph needs to locate and insert a cluster The specified distance between the inserted node and the existing center point is calculated. The insertion cluster split requires Calculate the specified distance value to determine the nearest neighbor center point set of the two new cluster centers after the split. times to calculate the specified distance value, so adding an inserted node to the corresponding cluster requires at most Calculate the specified distance value. Then, get the neighbor set of the inserted node. Calculate the specified distance value, and adjust the neighbor set of each neighbor of the inserted node. Calculate the specified distance value, so in Each layer of the layer neighbor graph needs to process the insertion of nodes times to calculate the specified distance value, so adding a single reference string requires a total of Calculates the specified distance value.

[0062] In one embodiment, when the operation type of the add, delete or modify instruction is deletion, a reference string to be deleted corresponding to the operation object is obtained. Performing the add, delete or modify operation on the index structure according to the add, delete or modify instruction specifically includes: The reference string to be deleted is marked as a deleted node, and the directed graph and the corresponding table containing the deleted node are processed layer by layer for the neighbor graph, which also specifically includes the update of the nearest neighbor center point set, the neighbor set and the corresponding table.

[0063] When processing a certain layer, the cluster where the deleted node is located is determined according to the corresponding table, the cluster where the deleted node is located is marked as a contracted cluster, and the deleted node is marked as a deleted node.

[0064] When all nodes in the shrunk cluster have deletion marks, execute the following processing steps.

[0065] Update the correspondence table: Delete all nodes in the contracted cluster from the directed graph, and delete the row where the center point of the contracted cluster is located from the correspondence table.

[0066] Update the nearest neighbor center point set of the center point: For each center point in the directed graph, if the nearest neighbor center point set of the center point contains the center point of the contraction cluster, then re-determine the center point in the directed graph with the smallest specified distance value to the center point. other center points, and update the center point's nearest neighbor center point set to be composed of these The set of other center points.

[0067] Update the node's nearest neighbor center point set: For each node in the directed graph, if the node's neighbor set contains at least one node in the contracted cluster, then re-obtain the center point of the cluster where the node is located, and all the center points in the nearest neighbor center point set of the center point of the cluster, for a total of center point. The set of center points is updated to the node’s nearest neighbor center point set.

[0068] Update neighbor set: For each node in the directed graph, if the node's neighbor set contains at least one node in the contracted cluster, then re-obtain the nearest neighbor center point set of the node in the cluster where all the center points are located, and the node with the smallest specified distance value Other nodes will The set of other nodes is updated to the neighbor set of the node to maintain the number of elements in the neighbor set of the neighbor to be .

[0069] If there are still nodes without deletion marks in the shrunk cluster, no further processing is performed.

[0070] The above incremental method of deleting a single reference string is In each layer of the layer neighbor graph, the contraction cluster can be directly located according to the correspondence table between the center point and its cluster, and the deleted node can be marked for deletion instead of directly deleting it from the multi-layer neighbor graph. This ensures that the index structure remains unchanged and does not require additional calculation of the specified distance value. The contraction cluster is deleted only if and only if the contraction cluster exists and all nodes in the contraction cluster have a deletion mark. The number of different center points that contain the center point of the contraction cluster in the center point's nearest neighbor center point set does not exceed , so a total of The specified distance value is calculated. The number of different nodes in the neighbor set that contain at least one node in the contracted cluster does not exceed , when updating the neighbor sets of these nodes, the updated nearest neighbor center points of these nodes need to be considered, and at most one of the original nearest neighbor center points of each of these nodes is replaced (that is, the center point of the contracted cluster is replaced by other centers), so it is only necessary to use all the nodes in the cluster where the other centers of the contracted cluster center are located (no more than ) to update the neighbor set, we can get the updated neighbor set of these nodes. That is, update these nodes that do not exceed The neighbor set of a node requires a total of In summary, if the deletion of a single reference string triggers the deletion of a shrinking cluster, a total of The specified distance value is not required, otherwise the specified distance value does not need to be calculated. Although the time cost of removing the entire contraction cluster is high, that is, it takes However, since this removal operation only occurs occasionally and can be completed independently by a background thread, it has little impact on string retrieval performance.

[0071] In one embodiment, add, delete, and modify operations are performed on an index structure according to add, delete, and modify instructions, including: when the operation type of the add, delete, and modify instructions is modification, obtaining a reference string before change and a reference string after change corresponding to the operation object; first marking the reference string before change as a deletion node to perform an add, delete, and modify operation with an operation type of deletion, and then marking the reference string after change as an insertion node to perform an add, delete, and modify operation with an operation type of addition.

[0072] In one embodiment, the specific execution method for the add, delete, and modify operations of the operation type of modification and the operation type of deletion has been described in detail above and will not be repeated here. Modifying a single reference string is equivalent to deleting the original reference string and adding the modified reference string. Therefore, modifying a single reference string requires no more than Calculates the specified distance value.

[0073] It is worth noting that in the search operations described above, the edit distance actually applicable may not be limited to the edit distance for only the three operations of addition, deletion and modification. The character modification operation can be subdivided into operations such as replacing characters with the same / similar pronunciation, replacing characters with similar glyphs, and replacing general characters, each of which can be set with a different unit cost. In addition, the applicable edit distance also allows the introduction of operations such as modifying words or phrases and exchanging two adjacent characters. The use of different edit distance settings should not be considered to exceed the technical scope protected by this application.

[0074] At the same time, this application does not describe a specific implementation method for the node modification operation during the index update process. The technical solution only points out that the node modification operation is equivalent to deleting the old node and adding a new node. This equivalence means that there are multiple specific feasible implementation methods, including (1) performing the deletion operation first and then the addition operation, (2) performing the addition operation first and then the deletion operation, and (3) first marking the deletion and then performing the addition operation, and finally performing the removal operation of the shrunk cluster under certain conditions, etc. These specific implementation methods are very intuitive and easy to think of, so limiting the specific implementation method should not be considered to exceed the technical scope protected by this application.

[0075] Step S130: According to the acquired search instruction, execute -Nearest neighbor search operation, return -The retrieval results output by the nearest neighbor retrieval operation.

[0076] In one embodiment, when a search instruction is obtained, the index structure is executed -Nearest neighbor search operation to achieve string search, that is, calculation The search instruction includes the query string, the expected result set size, and a search distance, wherein the search distance is a preset edit distance from one string to another, which is different from the specified distance used in the index building process mentioned later, but is an edit distance defined separately.

[0077] - The nearest neighbor search operation specifically includes: marking the query string as a test node, randomly selecting Nodes without deletion marks are marked as entry nodes of the first layer. The deletion marks of nodes are determined according to the index structure. The deletion marks are explained in detail in the subsequent operations of adding, deleting and modifying the index structure. We will not expand on them here. You only need to know that some nodes in the index structure will be marked with deletion marks. If the number of nodes without deletion marks in the first layer of the directed graph is less than , then all nodes without deletion marks are marked as entry nodes of the first-level directed graph.

[0078] Starting from the first layer of the neighbor graph, the processing operation is performed layer by layer until the first layer of the neighbor graph The processing operation includes: determining the After the entry node set of the layer directed graph is obtained, the entry node set is first marked as the node set at time 0 . Then according to the gradually increasing time , in turn by Time Node Set Determine Time Node Set ,Will The union of the neighbor sets of all nodes in is marked as the first union, and the first union is combined with The union of is marked as the second union, then Determine the node with the smallest retrieval distance to the test node in the second union If the number of nodes without deletion marks in the second union is less than , then The set is determined to be the set consisting of all nodes in the second union that are not marked for deletion.

[0079] It must converge in a finite time, when = When When converged, Mark as .

[0080] Determine whether the current number of processing layers has reached the maximum layer of the neighbor graph of the index structure: If , that is, the current layer is not the largest layer of the neighbor graph, then Marked as The entry node set of the layer directed graph and continue Correspondingly, if the current processing layer is the largest layer of the neighboring layers of the index structure, that is, when When Mark as search result and output.

[0081] The most expensive operation in the above string retrieval method is to calculate the edit distance between two strings, so the number of edit distance calculations can be used to measure the time complexity of the string retrieval method. Each layer in the layer neighbor graph needs to calculate Edit distance, where is the number of layers of the neighbor graph, is the number of nodes included in the search results, is the maximum number of nodes in a cluster, is the average number of iterations at each layer convergence, so the above string retrieval method needs to calculate a total of times the edit distance. In contrast, the original string retrieval method needs to calculate Edit distance, where is the number of reference strings. is small (usually between 1 and 9), and is much smaller than Therefore, the string retrieval method of this application can be regarded as the same as The time complexity of the unrelated constant-level approximate retrieval algorithm is significantly lower than the linear level of the original retrieval algorithm.

[0082] This application proposes to use incremental A method for accelerating the construction and updating of indexes by center point clustering. The key to the index structure of the layer neighbor graph is to limit the number of neighbors of each node, so that the retrieval process can be performed on the limited neighborhood of the entry node set rather than the entire set of nodes, thereby controlling the number of nodes traversed during the retrieval process. In order to limit the number of neighbors of the inserted node to a preset threshold The index construction method directly adapted to HNSW needs to find the node with the smallest specified distance value to the inserted node from all added nodes. nodes, so the index construction method directly adapted to HNSW requires calculation The distance value is specified once, while adding a node to the index requires calculation times the specified distance value, where The application maintains a corresponding table between each layer center point and its cluster and the center point's nearest neighbor center point set during index construction and incremental node addition to accelerate the search for the node with the smallest specified distance value from the inserted node from the added nodes. Specifically, when looking for this N When there are at most 10 nodes in the cluster, the technical solution of this application only needs to consider the number of nodes in the cluster where the nearest neighbor center point of the inserted node is located. nodes, and determining the nearest neighbor center point set of the inserted node only requires calculating The specified distance value between the cluster center point where the node is inserted and other centers, so adding a node to the index only requires calculating Specify the distance value. When constructing a neighbor graph of each layer, the technical solution of this application completes the calculation of the specified distance value between any two center points in each layer at one time, thus controlling the index construction process to The specified distance value is calculated. Much smaller than and is a large constant, so the time cost of the technical solution of the present application in the index construction process and the incremental node addition process is significantly lower than the time cost of directly adapting to HNSW.

[0083] In addition, the most expensive operation in the string retrieval method based on this index structure is the operation of calculating the edit distance between two strings, so the number of edit distance calculations can be used to measure the time complexity of the string retrieval method. Each layer in the layer neighbor graph needs to calculate Edit distance, where is the number of layers of the neighbor graph, is the number of nodes included in the search results, is the maximum number of nodes in a cluster, is the average number of iterations at each layer convergence, so the above string retrieval method needs to calculate a total of times the edit distance. In contrast, the original string retrieval method requires times (i.e. The linear time of the measure) edit distance calculation. Since in general is small (usually between 1 and 9), and is much smaller than Therefore, the string retrieval method of this application can be regarded as the same as The time complexity of the unrelated constant-level approximate retrieval algorithm is significantly lower than the linear level of the original retrieval algorithm.

[0084] Figure 2 The figure shows the relationship between the internal functional modules of a string search device in one embodiment. The string search device 20 includes: an index construction module 21, an index maintenance module 22 and a search module 23. The index construction module 21 is used to obtain a given reference string set to perform an index structure construction operation when there is no index structure. The index maintenance module 22 is used to generate an add, delete, or modify instruction according to the change of the reference string set when the reference string set corresponding to the index structure changes, and perform add, delete, or modify operations on the index structure according to the add, delete, or modify instructions. The search module 23 is used to perform a search on the index structure according to the obtained search instruction. -Nearest neighbor search operation, returns the search result output by the search operation.

[0085] Figure 3 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 3As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the string retrieval method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the string retrieval method. Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0086] In one embodiment, the present application further proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in any of the aforementioned embodiments.

[0087] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0088] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A character string retrieval method, characterized in that: The steps include: When the index structure does not exist, obtaining a given reference string set to perform an index structure construction operation; the index structure is composed of a neighbor graph and a correspondence table; The neighbor graph is given by Layer directed graph, is a preset integer; each layer of the directed graph is composed of nodes, and one node corresponds to one reference string; No. The directed graph stores all given reference strings; for not less than 1 and not greater than Any integer , No. The reference string set stored in the directed graph of the first layer is A subset of the reference string set stored in the directed graph; Part of the nodes of each layer of the directed graph are marked as center points, and the center points are determined according to the index structure construction operation. The center points divide all the nodes in the directed graph into multiple non-overlapping subsets, each of which is called a cluster, and the number of nodes in the cluster does not exceed , is a preset integer; in each layer of the directed graph, there are directed edges between some of the nodes, and the directed edges are determined according to the index structure construction operation; the nodes pointed by the directed edges are called neighbors of the nodes initiating the directed edges, and the number of neighbors of each node does not exceed ; Each layer of the neighbor graph has a unique corresponding table, and the corresponding table is a two-dimensional table consisting of rows and columns; each row corresponds to a center point of the corresponding layer in the neighbor graph; the columns have three columns: the first column stores the reference string corresponding to the center point; the second column stores the reference string corresponding to other nodes in the cluster where the center point is located except the center point; the third column stores the reference string set corresponding to the nearest neighbor center point set of the center point, and the nearest neighbor center point set of the center point is determined by the index structure construction operation; When the reference string set corresponding to the index structure changes, generating an add, delete, or modify instruction according to the change of the reference string set, and performing add, delete, or modify operations on the index structure according to the add, delete, or modify instruction; According to the retrieved retrieval instruction, execute -Nearest neighbor search operation, returning the - The search result output by the nearest neighbor search operation; the search instruction content includes the query string, the expected result set size and a search distance, wherein the search distance is a preset edit distance from one character string to another character string; Said -Nearest neighbor search operations include: Mark the query string as a test node; randomly select A node without a deletion mark is marked as the entry node of the first layer. The deletion mark of the node is determined according to the index structure. If the number of nodes without a deletion mark in the directed graph of the first layer is less than , then all nodes without deletion marks are marked as entry nodes of the directed graph described in the first layer; The processing operation is performed layer by layer starting from the first layer of the neighbor graph until the first layer of the neighbor graph The layer processing is completed, and the processing operation includes: In determining the After the entry node set of the directed graph is obtained, the entry node set is first marked as the node set at time 0. ; Then according to the gradually increasing time , in turn by Time Node Set Determine Time Node Set ,Will The union of the neighbor sets of all nodes in is marked as the first union, and the first union is combined with The union of is marked as the second union, then Determine the node with the smallest search distance to the test node in the second union The number of nodes without deletion marks in the second union is less than , then Determine a set consisting of all nodes without deletion marks in the second union; when = Time will Mark as ; like , then Marked as The entry node set of the directed graph at the layer and continue Layer processing; like , then Mark as the search result and output.

2. The character string search method according to claim 1, wherein: The index structure building operation includes: Determine the minimum level at which each reference string is added to the neighbor graph l Probability of layer : ； is a preset real number greater than 1; according to Randomly determine the minimum level at which each given reference string is added to the neighbor graph l , and add the given reference strings as nodes to the first node of the neighbor graph one by one Layer to In the directed graph of layers; When adding a corresponding node of the reference string in each layer of the directed graph, marking the node to be added as an insertion node; The first inserted node is regarded as the first center point and forms a cluster by itself; subsequently, in the insertion process, the specified distance value between the newly inserted node and the existing center point is calculated one by one, and the specified distance value is a preset literal distance measurement value between the two nodes that satisfies non-negativity, symmetry and triangle inequality; Marking the cluster where the center point with the smallest specified distance value is located as an insertion cluster, and adding the insertion node to the insertion cluster; When the number of nodes in the inserted cluster exceeds , split the inserted cluster into two new clusters, the center points of the two new clusters are two different nodes in the inserted cluster, satisfying that the sum of the deviation values ​​from other nodes in the inserted cluster to the two new clusters is the minimum, and the deviation value of each node to the two new clusters is the minimum value of the specified distance value between the node and the center points of the two new clusters; the other nodes in the extended cluster except the center points of the two new clusters are called to-be-processed nodes, and the specified distance values ​​between the to-be-processed nodes and the center points of the two new clusters are calculated; the to-be-processed nodes are added to the new cluster with the smaller specified distance value; if the specified distance values ​​between the to-be-processed nodes and the center points of the two new clusters are the same, then the to-be-processed nodes are added to the new cluster with fewer nodes; After all given reference strings are added as nodes to the neighbor graph, the clusters included in each layer of the directed graph of the neighbor graph are determined; the node with the smallest sum of specified distance values ​​from other nodes in each cluster is calculated and marked as the center point of each cluster; For satisfaction Every integer , according to the neighbor graph All the center points and other nodes in the layer determine the neighbor relationship to build the first the correspondence table of layers; The neighbor graph and the corresponding table of each layer thereof are aggregated to obtain the index structure.

3. The character string search method according to claim 2, wherein: The neighbor graph is All the center points and other nodes in the layer determine the neighbor relationship to build the first The corresponding table of layers includes: Calculate the The specified distance value between the center points in the layer; according to the specified distance value between the center points, determine for each center point other center points with the smallest specified distance value, and The set of other center points is marked as the nearest neighbor center point set of the center point; For the l For each node in the layer, obtain the center point of the cluster where the node is located, and all the center points in the nearest neighbor center point set of the center point of the cluster, a total of center point; The set of center points is marked as the nearest neighbor center point set of the node; Get the cluster of all the center points in the nearest neighbor center point set of each node, with the smallest distance value to the node. other nodes, The set of other nodes is marked as the neighbor set of the node; Add the node to the directed graph of the layer directed edges to other nodes; According to The reference string corresponding to each of the center points in the layer, the reference strings corresponding to other nodes in the cluster where the center point is located, and the set of nearest neighboring center points of the center point are determined. The correspondence table of the layer.

4. The character string search method according to claim 1, wherein: The add, delete and modify instruction includes an operation type and an operation object, and the operation type is one of add, delete and modify; When the operation type is adding, the operation object is the reference string to be added; When the operation type is deletion, the operation object is the reference character string to be deleted; When the operation type is modification, the operation objects are the reference character string before modification and the reference character string after modification.

5. The character string search method according to claim 4, wherein: The performing the add, delete, and modify operations on the index structure according to the add, delete, and modify instructions includes: When the operation type of the add, delete or modify instruction is add, obtaining a reference string to be added corresponding to the operation object, and marking the reference string to be added as an insert node; Determine the minimum level at which the inserted node is added to the neighbor graph Probability of layer : ； is a preset real number greater than 1; according to Randomly determine the minimum layer to which the inserted node is added , and add the inserted node to the neighbor graph Layer to In the directed graph of layers; When adding the inserted node to each layer of the directed graph, calculating the specified distance value between the inserted node and all the center points in the directed graph, marking the cluster where the center point with the smallest specified distance value is located as an extended cluster, and adding the inserted node to the extended cluster; When the number of nodes in the extended cluster after adding the inserted node exceeds , split the extended cluster into two new clusters, the center points of the two new clusters are two different nodes in the extended cluster, so that the sum of the deviation values ​​of other nodes in the extended cluster to the two new clusters is minimized, and the deviation value of each node to the two new clusters is the minimum value of the specified distance value between the node and the center points of the two new clusters; the other nodes in the extended cluster except the center points of the two new clusters are called to-be-processed nodes, and the specified distance value between the to-be-processed node and the center points of the two new clusters is calculated; the to-be-processed node is added to the new cluster with the smaller specified distance value; if the specified distance value between the to-be-processed node and the center points of the two new clusters is the same, then the to-be-processed node is added to the new cluster with fewer nodes; Calculate the specified distance value between the center point of the two new clusters and other center points, and determine the center point of each new cluster with the minimum specified distance value according to the specified distance value. other center points, The set of other center points is marked as the nearest neighbor center point set of the center point of the new cluster; For each node in the two new clusters, obtain the center point of each node corresponding to the new cluster and all the center points in the nearest neighbor center point set of the center point of the new cluster, a total of center point; The set of center points is marked as the nearest neighbor center point set of each node in the new cluster; Get the cluster of all the center points in the nearest neighbor center point set of each node, with the smallest distance value to the node. other nodes, The set of other nodes is marked as the neighbor set of the node; Deleting the row where the center point of the extended cluster is located from the correspondence table, and adding corresponding row information of the center points of the two new clusters to the correspondence table according to the node sets, center points of the two new clusters and the nearest neighbor center point sets of the center points of the new clusters; Mark each node in the neighbor set of the inserted node that does not belong to any of the new clusters as a neighbor to be processed, first add the inserted node to the neighbor set of the neighbor to be processed, and then delete the neighbor with the largest specified distance value from the neighbor set of the neighbor to be processed; When the number of nodes in the extended cluster after adding the inserted node does not exceed , obtain the center point of the cluster where the inserted node is located, and all the center points in the nearest neighbor center point set of the center point of the cluster, a total of center point; The set of center points is marked as the nearest neighbor center point set of the inserted node; Get the cluster of all the center points in the nearest neighbor center point set of the inserted node, which has the smallest specified distance value from the inserted node other nodes; The set consisting of other nodes is marked as the neighbor set of the inserted node; The second column information of the row where the center point of the extended cluster is located in the corresponding table is updated according to the node set of the extended cluster; each node in the neighbor set of the inserted node is marked as a neighbor to be processed; the inserted node is first added to the neighbor set of the neighbor to be processed, and then the neighbor with the largest specified distance value is deleted from the neighbor set of the neighbor to be processed.

6. The character string search method according to claim 4, wherein: The performing the add, delete, and modify operations on the index structure according to the add, delete, and modify instructions includes: When the operation type of the add, delete or modify instruction is deletion, obtaining a reference string to be deleted corresponding to the operation object, and marking the reference string to be deleted as a deletion node; Processing the directed graph and the corresponding table containing the deleted node layer by layer for the neighbor graph; determining the cluster where the deleted node is located according to the corresponding table, marking the cluster where the deleted node is located as a contracted cluster, and marking the deleted node with a deletion mark; When all nodes in the contracted cluster have deletion marks, all nodes in the contracted cluster are deleted from the directed graph, and the row where the center point of the contracted cluster is located is deleted from the corresponding table; For each center point in the directed graph, if the center point of the shrinking cluster is included in the set of the nearest neighboring center points of the center point, the center point in the directed graph with the smallest specified distance value to the center point is re-determined. other center points, and update the nearest neighbor center point set of the center point to be composed of these The set of other center points; For each node in the directed graph, if the neighbor set of the node includes at least one node in the contracted cluster, the center point of the cluster where the node is located and all the center points in the nearest neighbor center point set of the center point of the cluster where the node is located are re-obtained, and a total of center point; The set of center points is updated as the nearest neighbor center point set of the node; the cluster of all center points in the nearest neighbor center point set of the node with the smallest specified distance value to the node is re-obtained. other nodes, The set consisting of other nodes is updated as the neighbor set of the node; When there are still nodes without deletion marks in the contracted cluster, no further processing is performed.

7. The character string search method according to claim 4, wherein: The performing the add, delete, and modify operations on the index structure according to the add, delete, and modify instructions includes: When the operation type of the add, delete or modify instruction is modification, obtaining a reference string before modification and a reference string after modification corresponding to the operation object; First, the reference string before the change is marked as a deletion node to perform the addition, deletion and modification operation with the operation type of deletion, and then the reference string after the change is marked as an insertion node to perform the addition, deletion and modification operation with the operation type of addition.

8. A character string search device, characterized in that: include: An index building module, used for obtaining a given reference string set to perform an index structure building operation when no index structure exists; An index maintenance module, configured to generate an add, delete, or modify instruction according to the change of the reference string set when the reference string set corresponding to the index structure changes, and perform an add, delete, or modify operation on the index structure according to the add, delete, or modify instruction; A retrieval module is used to execute a search for the index structure according to the obtained search instruction. -Nearest neighbor search operation, returning the search result output by the search operation.

9. A computer device, characterized in that: including a processor and a memory; The processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images