Fast graph search method for fast inner product search on graphs

Abstract

This paper presents implementations of a fast search method on graphs for Maximum Inner Product Search (MIPS). This optimization problem is challenging because conventional approximate nearest neighbor (ANN) search methods cannot be performed efficiently on non-metric similarity measures. The implementations in this paper introduce the properties of isomorphism between subgraphs of the #imgabs2# graph and the Delaunay graph for inner products based on the #imgabs0# / class #imgabs1# transformation. Based on this observation, a novel implementation of graph indexing and search methods is proposed to find the optimal solution with the maximum inner product with the query. Experiments show significant improvement over existing methods.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application is related to a pending and commonly owned patent application filed on November 1, 2019 entitled “ TRANSFORMATION FOR FAST INNER PRODUCT SEARCH ON GRAPH The present invention is related to and claims priority from U.S. patent application Ser. No. 62 / 929,751 (Docket No. 28888-2367P) listing Shulong Tan, Zhixin Zhou, Zhaozhuo Xu and Ping Li as inventors, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure generally relates to systems and methods for computer learning that can provide improved computer performance, features, and uses. More specifically, the present disclosure relates to systems and methods for fast searching. Background Art

[0004] Deep learning (DL) has been widely used in many different fields, such as computer vision, natural language processing, medical image processing, etc. Deep learning has gradually laid the foundation for the relatively recent artificial intelligence / machine learning (AI / ML) revolution, which has affected many industries and spawned more industries. In many applications, it is important to perform searches quickly.

[0005] Therefore, what are needed are systems and methods for fast searching on data sets that include high-dimensional representations, such as high-dimensional vectors. Summary of the invention

[0006] In a first aspect, the present disclosure provides a computer-implemented method, comprising: forming a transformed data set by transforming a data set including an original data element and by adding the original element to the transformed data set; using l 2 The invention relates to a method for constructing an approximate graph of a transformed data set, wherein at least some of the data elements of the transformed data set are vertices in the approximate graph and connections between vertices in the approximate graph are boundaries; identifying a set of neighbors of the original element on the approximate graph; removing the original element and its boundaries from the approximate graph; obtaining a final graph by replacing vertices in the approximate graph with corresponding original data elements, the approximate graph including data elements from the transformed data set; and outputting the final graph for searching using inner product as a ranking measure.

[0007] In a second aspect, the present disclosure provides a non-transitory computer-readable medium or media comprising one or more instruction sequences, which, when executed by at least one processor, cause the execution of operations comprising the following steps: forming a transformed data set by transforming a data set comprising original data elements and by adding the original elements to the transformed data set; using l 2 The invention relates to a method for constructing an approximate graph of a transformed data set, wherein at least some of the data elements of the transformed data set are vertices in the approximate graph and connections between vertices in the approximate graph are boundaries; identifying a set of neighbors of the original element on the approximate graph; removing the original element and its boundaries from the approximate graph; obtaining a final graph by replacing vertices in the approximate graph with corresponding original data elements, the approximate graph including data elements from the transformed data set; and outputting the final graph for searching using inner product as a ranking measure.

[0008] In a third aspect, the present disclosure provides a system, comprising: one or more processors; and a non-transitory computer-readable medium or media, comprising one or more instruction sequences, which, when executed by at least one of the one or more processors, cause the execution of operations comprising the following steps: forming a transformed data set by transforming a data set comprising original data elements and by adding the original elements to the transformed data set; using l 2 The invention relates to a method for constructing an approximate graph of a transformed data set, wherein at least some of the data elements of the transformed data set are vertices in the approximate graph and connections between vertices in the approximate graph are boundaries; identifying a set of neighbors of the original element on the approximate graph; removing the original element and its boundaries from the approximate graph; obtaining a final graph by replacing vertices in the approximate graph with corresponding original data elements, the approximate graph including data elements from the transformed data set; and outputting the final graph for searching using inner product as a ranking measure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Reference will be made to embodiments of the present disclosure, examples of which may be shown in the accompanying drawings. These drawings are intended to be illustrative and not restrictive. Although the present disclosure is generally described in the context of these embodiments, it should be understood that the scope of the present disclosure is not intended to be limited to these particular embodiments. Items in the drawings may not be drawn to scale.

[0010] Figure 1 A method for generating a graph and searching using the graph according to an embodiment of the present disclosure is described.

[0011] FIG. 2A to FIG. 2D Depicted are (A) an empty half-space criterion for an IP-Delaunay graph; (B) an IP-Delaunay graph; and (C) a transformed half-space criterion for an IP-Delaunay graph according to an embodiment of the present disclosure.2 -Delaunay map empty sphere criterion; and (D) the transformed l 2 -Delaunay graph. The boundary 205 forms a subgraph isomorphic to the IP-Delaunay graph.

[0012] Figure 3 A method of performing a greedy search according to an embodiment of the present disclosure is described.

[0013] Figure 4 An alternative method for graph construction according to an embodiment of the present disclosure is depicted.

[0014] Figure 5 A method of performing neighborhood selection according to an embodiment of the present disclosure is described.

[0015] Figure 6 An overall method according to an embodiment of the present disclosure is depicted.

[0016] Figure 7 The experimental results of (top 1) calls and queries per second on different data sets according to embodiments of the present disclosure are depicted. The top curve shows the superiority of the corresponding method. Figures, ip-NSW, ip-NSW-nohie, ip-NSW-no-sel represent the implementations of the proposed method, ip-NSW with both hierarchy and boundary selection, ip-NSW without hierarchy, and ip-NSW without boundary selection, respectively.

[0017] FIG. 8A to FIG. 8C Depicts experimental results of calls and queries per second on different datasets according to embodiments of the present disclosure. Focus is on the first, tenth, and hundredth ground truth labels. In addition, the best result is in the upper right corner.

[0018] Fig. 9 Depicts experimental results of call and compute percentage on different data sets according to embodiments of the present disclosure. The best result is in the lower right corner.

[0019] Fig.10 Describes an embodiment of the present disclosure. The figure shows a comparison between the implementation scheme and two implementation schemes of HNSW and SONG on ip-NSW.

[0020] FIG. 11A to FIG. 11C Describes an embodiment of the present disclosure Fig. 2 Experimental results of ip-NSW with and without boundary selection.

[0021] FIG. 12A to FIG. 12C Describes an embodiment of the present disclosure Figure 1. Implementation method, using ip-NSW with random initialization points and using Experimental results of ip-NSW at the initial point of Fig.

[0022] FIG. 13A to FIG. 13C Describes an embodiment of the present disclosure Implementation scheme and experimental results of ip-NSW with and without hierarchical structure.

[0023] FIG. 14A to FIG. 14B Experimental results of call and compute percentages according to embodiments of the present disclosure are depicted.

[0024] Fig.15 Depicted is a simplified block diagram of a computing device / information processing system in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In the following description, for the purpose of explanation, specific details are set forth in order to provide an understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these details. In addition, those skilled in the art will recognize that the embodiments of the present invention described below may be implemented in various ways (e.g., processes, devices, systems, equipment, or methods) on a tangible computer-readable medium.

[0026] The components or modules shown in the accompanying drawings are exemplary illustrations of embodiments of the present invention and are intended to avoid obscuring the present invention. It should also be understood that throughout this discussion, components may be described as separate functional units (which may include subunits), but those skilled in the art will recognize that various components or parts thereof may be divided into separate components, or may be integrated together (including, for example, into a single system or component). It should be noted that the functions or operations discussed herein may be implemented as components. Components may be implemented in software, hardware, or a combination thereof.

[0027] In addition, the connection between the components or systems in the drawings is not intended to be limited to direct connection. On the contrary, the data between these components can be modified, reformatted, or otherwise changed by the intermediate components. In addition, additional or fewer connections can be used. It should also be noted that the terms "connect", "connect", or "communicatively connected", "engaged with", "engaged" or any of their derivatives should be understood to include direct connection, indirect connection and wireless connection performed by one or more intermediate devices. It should also be noted that any communication (e.g., signal, response, reply, reply, message, inquiry, etc.) can include the exchange of one or more information.

[0028] References to "one or more embodiments", "preferred embodiments", "an embodiment", "an embodiment", etc. in this specification indicate that the specific features, structures, characteristics or functions described in conjunction with the embodiments are included in at least one embodiment of the present disclosure, and may be included in more than one embodiment. In addition, the appearance of the above-mentioned phrases in various places in this specification does not necessarily all refer to the same embodiment or multiple identical embodiments.

[0029] The use of certain terms in various places in this specification is intended to be illustrative and should not be construed as limiting. Services, functions, or resources are not limited to a single service, a single function, or a single resource; the use of these terms may refer to a distributable or aggregated grouping of related services, functions, or resources. The terms "include," "comprises," "includes," and "includes" should be understood as open terms, and any listings thereafter are examples and are not intended to be limited to the listed items. A "layer" may include one or more operations. The words "optimized," "optimize," "optimize," and the like refer to improvements in results or processes and do not require that a specified result or process reach an "optimal" or peak state. Memory, database, information base, data storage, table, hardware, cache, and the like may be used herein to refer to system components or components in which information may be input or otherwise recordable.

[0030] In one or more embodiments, the stopping conditions may include: (1) a set of iterations has been executed; (2) a processing time amount has been reached; (3) convergence (such as the difference between consecutive iterations is less than a first threshold); (4) divergence (such as performance deterioration); and (5) an acceptable result has been achieved.

[0031] Those skilled in the art will recognize that: (1) certain steps may be performed optionally; (2) the steps may not be limited to the specific order described herein; (3) certain steps may be performed in a different order; and (4) certain steps may be performed simultaneously.

[0032] Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims. Each reference mentioned in this patent document is incorporated herein by reference in its entirety.

[0033] It should be noted that any experiments and results provided herein are provided by way of illustration and were performed under specific conditions using a specific embodiment or embodiments; therefore, neither these experiments nor their results should be used to limit the scope of the current patent document disclosure.

[0034] A. General Introduction

[0035] One of the focuses of this paper is discrete optimization problem. Given a large dataset S with high-dimensional vectors and a query point q in Euclidean space, the goal is to find the inner product Maximize x∈S. Strictly speaking, the following implementation of an efficient method for computing is developed:

[0036]

[0037] This so-called Maximum Inner Product Search (MIPS) problem has wide applicability in machine learning models such as recommender systems, natural language processing and multi-class or multi-label classifiers, computational advertising for search engines, etc. Due to the importance and popularity of the MIPS algorithm, effective and efficient MIPS algorithms have been extensively studied.

[0038] Early approaches proposed tree-based methods to solve the MIPS problem. Recently, a series of works in the literature have attempted to transform MIPS into a conventional approximate nearest neighbor (ANN) search by asymmetrically lifting the basic data vectors and query vectors into a high-dimensional space. After this transformation, well-developed ANN search methods can then be applied to solve the MIPS problem. There are other proposals designed for the MIPS task, including quantization-based methods and graph-based methods.

[0039] In this patent document, a new implementation of a graph-based approach to MIPS is proposed. Graph-based approaches have been developed for ANN search in metric space and have shown significant superiority. Recent work (i.e., ip-NSW) attempts to extend graph-based approaches for ANN search to MIPS. The authors of the ip-NSW approach introduced the concept of an IP-Delaunay graph, which is the minimum graph that can guarantee the return of an exact solution to MIPS through a greedy search. In practice, ip-NSW attempts to approximate the IP-Delaunay graph via navigable small worlds (NSW) and hierarchical navigable small worlds (HNSW). In order to extend beyond existing approaches, an implementation of a graph-based approach to MIPS is proposed that retains the advantages of similarity graphs in metric space.

[0040] The implementation in this paper is based on a transformation on a dataset that connects graph-based indexes for MIPS and ANN search. or class Under the transformation, there is an isomorphism between the two graphs: (a) IP-Delaunay graph before the transformation; and (b) after the transformation with respect to l 2 -norm Delaunay triangulation subgraph (l 2Based on this observation, in one or more embodiments, the IP-Delaunay graph can be approximated in two steps in one or more embodiments: (i) via kind transform the mapped data points; and (ii) approximate l at the transformed data points and an additional point for the origin 2 - Delaunay graph. Then, in one or more embodiments, given a query point, a greedy search may be performed on the obtained graph by comparing the inner product of the query with the data points (nodes / vertices in the graph) in their original format.

[0041] The embodiments discussed herein are advantageous due to at least two advantages: (a) based on l 2 -distance graph construction maintains all the advantageous features of similar graphs in metric space; and (b) the additional point (i.e., the origin) will be connected to different high-norm points (usually MIPS solutions), which naturally provides a good starting point for greedy search. Empirical experiments have shown that these features significantly improve efficiency.

[0042] B. Graph-based search methods and search implementation methods

[0043] Graph-based search methods typically first construct a well-designed similarity graph (e.g., a kNN graph in an approximate nearest neighbor (ANN) search) and then perform a greedy search on the graph. A simple greedy search (such as that used for the maximum inner product search (MIPS) task) can be described as follows. Given a graph and a query, an implementation of the method randomly selects vertices from the graph and then estimates the inner product of the query with a random seed vertex and the vertex's neighborhood. If one of its neighbors has a larger inner product with the query than the vertex itself, the neighborhood is treated as a new seed vertex and the search steps are repeated. The process stops when it is found that the inner product of a vertex with the query is larger than all of the vertex's neighbors. Greedy search has a generalized version, which will be introduced in Method 1 below.

[0044] It has been pointed out that in order to find an exact solution to the nearest neighbor search or MIPS by a greedy search strategy, the graph should contain the Delaunay graph of the search measure (see Definition 2 below) as a subgraph. 2 -distance common ANN search case, the index map should contain information about l 2 -The Delaunay graph of the distance (called l 2 -Delaunay graph) as a subgraph. In practice, since it is difficult to construct an exact Delaunay graph, an approximate l 2-Delaunay graphs, such as VoroNet and Navigable Small Worlds (NSW). Based on NSW, Hierarchical NSW (HNSW) networks exploit the hierarchical graph structure and heuristic boundary selection criteria (see Method 3 below for details), and often achieve performance improvements in ANN search tasks.

[0045] In one or more embodiments, the idea of ​​the Delaunay graph can be extended to the inner product. The best graph to get the exact MIPS by a simple greedy search is the Delaunay graph about the inner product (called the IP-Delaunay graph). Stanislav Morozov and Artem Babenko disclosed ip-NSW in "Non-metric similarity graphs for maximum inner product search" on pages 4726 to 4735 of "Progress in Neural Information Processing Systems (NeurIPS)" in Montreal, Canada (2018), which attempts to extend HNSW for metric spaces to MIPS. It is worth noting that the authors of the paper cited above show some important properties of the Delaunay graph. However, their HNSW-based graph construction algorithm for the inner product has some shortcomings, including:

[0046] (1) Since the boundary selection criterion of HNSW is not applicable to inner products, the associated boundaries of vertices may have very similar directions, which reduces efficiency.

[0047] (2) The hierarchical graph structure of HNSW is helpful for metric ANN search, but has little impact on the MIPS problem.

[0048] These claims are validated by experiments comparing with different versions of ip-NSW. The effect of boundary selection can be positive or negative in different datasets. The hierarchical structure does not change the efficiency of inner product search. To address the boundary selection problem, previously jointly assigned U.S. patent application No. 17 / 023,283, filed on September 16, 2020, entitled “EFFICIENT RETRIEVAL OF TOPSIMILARITY REPRESENTATIONS,” which lists Shulong Tan, Zhixin Zhou, Zhaozhuo Xu, and Ping Li as inventors (which claims priority to U.S. patent application No. 62 / 923,459, filed on October 18, 2019, entitled “EFFICIENT RETRIEVAL OF TOP SIMILARITY REPRESENTATIONS,” which lists Shulong Tan, Zhixin Zhou, Zhaozhuo Xu, and Ping Li as inventors (the documents of which are incorporated herein by reference in their entirety)), discloses an implementation of an appropriate boundary selection method for inner products, generally referred to as IPDG. IPDG implementations significantly improve upon previous MIPS, but some implementations show some potential performance limitations to the top n (n>1) results. In this patent document, the IP-Delaunay graph for MIPS (referred to as Figure / Class Implementation of different approximations in FIG. 7 provides state-of-the-art MIPS methods for various top-n MIPS results.

[0049] One intuition is that if we can find an approximate graph that maps the IP-Delaunay graph in the original space to the transformed space (e.g., 2 -Delaunay graph), we can make full use of the successful l 2 -Delaunay graph approximation method to construct IP-Delaunay graph. Given each data point x i , perform the transformation y i :=x i / ‖x i ‖ n (Here n>1 and n=2 is Transformation), which produces a new dataset: After the transformation, existing graph construction methods (e.g., HNSW or SONG implementations (SONG implementations are described in commonly assigned U.S. patent application Ser. No. 62 / 940,043, filed on Nov. 25, 2019, entitled “APPROXIMATE NEAREST NEIGHBOR SEARCH FOR PARALLEL PROCESSING HARDWARE COMPONENTS,” which lists Weijie Zhao, Shulong Tan, and Ping Li as inventors (the documents of which are incorporated herein by reference in their entirety))) can be applied, and the transformed data (e.g., ) (e.g., approximation l 2 -Delaunay graph). It is found that the IP-Delaunay graph of S is the same as that of The neighborhood of 0 in the approximate graph of isomorphism. Details on this statement can be found in Section C below. In short, one or more implementations can be summarized in the following steps and Figure 1 Depicted in:

[0050] 1. Form (105) a transformed data set by performing a transformation. For example, in one or more embodiments, is the transformed data set, where n>1 (and when n=2 is transform).

[0051] 2. Construct (110) about the transformed data set The approximate graph (for example, approximate l 2 -Delaunay diagram, (e.g., HNSW)).

[0052] 3. Identify (115) a set of neighborhoods N of the original element 0 on the approximate graph. For example, let N denote the neighborhood of 0 in the graph from the previous step. Then, in one or more embodiments, remove 0 and its associated boundaries from the graph and replace them with the original data vector x i Replace vertex y i , the final graph (120) can be obtained.

[0053] 4. Output (125) the final graph that can be used for searching. For example, in one or more embodiments, let N be the initial vertices, and then, for the query element, perform a greedy inner product search on the graph, where the inner product is used as the ranking measure.

[0054] It should be noted that in one or more embodiments, the greedy search method starts from a set of initial points rather than from data point 0, because 0 is not in S. In the generalized greedy search described in method 1, there may be multiple initial points. The equivalent description starts from 0, but never returns it as a result. Compared with the existing graph-based search method for MIPS (i.e., ip-NSW), the embodiments of this invention use l 2 -distance (on the transformed data) to construct an index graph that largely retains the advantageous features of the metric similarity graph. In addition, the embodiment starts the search from carefully selected different top norm points N (its usage is similar to the hierarchical structure of HNSW), resulting in more efficient performance. Therefore, the embodiment largely overcomes the weaknesses of existing graph-based search methods, and it is not surprising that the embodiment performs better empirically.

[0055] C. Transformation and Delaunay graph isomorphism implementation

[0056] In order to find the exact nearest neighbor by simple greedy search, the graph should contain Delaunay graphs as subgraphs. This formulation can be extended to the MIPS problem. For generality, for any continuous binary function Voronoi cells and Delaunay diagrams may be introduced; however, one or more embodiments herein are interested in the inner product and negative l 2 -The case where the norm f(x,y) = -‖xy‖.

[0057] Definition 1: For a fixed And given the function f, the Voronoi cell R i Can be limited to:

[0058]

[0059] In one or more embodiments, the Voronoi cells determine the solution to the MIPS problem. From the above definition, it can be observed that when If and only if q∈R j hour, Because recording Voronoi cells can be very expensive, we can record their dual graph instead, the Delaunay graph, which is defined as follows.

[0060] Definition 2: For a fixed function f and a data set And given the Voronoi cell R with respect to f and S i , i = 1, 2, ..., n, the Delaunay graph is an undirected graph with vertices S and a boundary {x i ,x j} if and only if Existence at the time.

[0061] The Delaunay diagram records the adjacency relationship of the Voronoi cells. i and unit R j are adjacent to each other, then their corresponding nodes x i and x j If f(x,y) = -‖xy‖, then the graph is called l 2 -Delaunay graph. If Then the graph is called an IP-Delaunay graph.

[0062] In one or more embodiments, the scope of the MIPS problem can now be reduced. as well as And the purpose is to solve the optimization problem of formula 1. In one or more embodiments, 0 is changed from First, 0 has the same inner product value as any point. Second, if 0 is not removed, each Voronoi cell about the inner product contains 0 as a common element, so the Delaunay graph will be fully connected and uninteresting. In one or more embodiments, the following mild assumptions can be made on the data set to simplify the analysis.

[0063] Assumption 1: The dataset S satisfies that its conical shell is the entire space. More precisely,

[0064]

[0065] Assumption 2 (General Position): For k = 2, 3, ..., d + 1, there are no k points of the dataset S on the (k-2)-dimensional affine hyperplane, or no k + 1 points of S on any (k-2)-dimensional sphere. If so, the dataset S is said to be in a general position.

[0066] Assumptions 1 and 2 are usually mild in real data. When the data points are embedding vectors of users, items, entities (in recommender systems), or sentences (in natural language processing). In this case, the entries of the data vectors are distributed over the entire real line. With high probability, each hyperoctant contains at least one data point such that the convex hull of the dataset contains 0 as an interior point. If the data vectors in S independently and identically obey For any continuous distribution on , the probability of hypothesis 2 being true is 1. For such a data set S, the corresponding l 2 -Delaunay diagrams and IP-Delaunay diagrams are unique. Two important criteria for these Delaunay diagrams can now be introduced.

[0067] Proposition 1 (Empty Half-Space Criterion): For a fixed data set Assume existence The open half space H satisfies: (a)x i and x j On the boundary line of H, (b) H does not contain data points, then there is a connection x in the IP-Delaunay graph i and x j Conversely, if such a boundary exists, then the open half-space H must exist.

[0068] In other words, the empty half-space criterion states that in the IP-Delaunay graph, the boundary {x i ,x j} exists, the hyperplane passes through x i and x j , so that one of its corresponding open half-spaces is empty and the other contains all the elements except x i and x j All data points outside the IP-Delaunay graph. The empty half-space criterion is related to l 2 -The empty sphere criterion for Delaunay plots is closely related and is shown below.

[0069] Proposition 2 (Empty Sphere Criterion): For a fixed data set A subset of d+1 points of S is in the l corresponding to S if and only if the circumscribed sphere of these points does not contain any other point from the dataset S inside the sphere. 2 -Delaunay graph is fully connected.

[0070] Once this criterion is satisfied, the subgraph of d+1 vertices is called a d-simplex. The connection between these criteria can be demonstrated by the transformation

[0071]

[0072] Under this transformation, each hyperplane can be mapped to a sphere passing through the origin. This is because The transformation on has the following facts:

[0073]

[0074] For an orthogonal matrix A and ∈ = 0 or 2, Indeed, according to Liouville's conformal mapping theorem, for d>2, formula (3) describes all Transformation. An important and useful property of the transformation states that if the hyperplane does not pass through the origin, then it The transformed images are all spheres passing through the origin.

[0075] FIG. 2A to FIG. 2D An example when d=2 is shown. Figure 2A The line AB in the plane divides the plane into two open half-spaces. One of the half-spaces does not contain any data points, so A and B are connected in the IP-Delaunay graph by Proposition 1. Let A′, B′ be the transformed images of A and B in formula (2). According to The properties of the transformation, the image of line AB is Figure 2C The circumscribed circle of the midpoints 0, A′, and B′. The empty half-space criterion for A and B implies that the circumscribed circle does not contain any data points, so according to the empty sphere criterion, l 2 -There exists a simplex in the Delaunay graph with vertices 0, A′ and B′. This observation can be formalized as follows:

[0076] Theorem 1: Assume Assume that S satisfies Assumptions 1 and 2. For i∈[n], let y i :=x i / ||x i || 2 , S′:={y1,…,y n}as well as Then the following are equivalent:

[0077] (a) The IP-Delaunay graph of S contains the boundary {x i ,x j}.

[0078] (b) Existence Make

[0079] (c) There exists c∈X such that

[0080] (d) In relation to l 2 -There exists a d-simple in the Delaunay graph containing the vertices {0,y i ,y j}.

[0081] The equivalence between (a) and (d) in the theorem implies that the IP-Delaunay graph is related to l 2 -The isomorphism relationship between the subgraphs of the Delaunay graph. Therefore, people immediately have the next inference.

[0082] Corollary 1: The following graphs are isomorphic after removing their isolated vertices:

[0083] (a) IP-Delaunay graph on S,

[0084] (b) Satisfy the following conditions On each boundary {y i ,y j} 2 -Subgraph of the Delaunay graph: 2 -There exists a d-simple in the Delaunay graph containing the vertices {0,y i ,y j},

[0085] Here the isomorphism relation is for x i of It is not isolated in the IP-Delaunay graph.

[0086] consider FIG. 2A to FIG. 2D In the example, Corollary 1 says, Figure 2B The IP-Delaunay graph in Figure 2D The dashed line 205D subgraph isomorphic. Therefore, l 2 The good approximation of the -Delaunay graph also applies to the approximation of the IP-Delaunay graph. The next section includes implementation details.

[0087] Remark 1 (Convex hull and extreme points): If a vertex is not isolated in the IP-Delaunay graph, it can be said to be an extreme point. Under Assumption 1, a point is an extreme value if and only if it lies on the boundary line of the convex hull of S. In this case, constructing the IP-Delaunay graph is equivalent to finding the convex hull. In Corollary 1, an equivalent method for finding the convex hull of a finite set is derived. For the purpose of constructing the convex hull, Assumption 1 is not necessary because it always holds after some transformations. It is worth noting that there is a method for finding the convex hull. This method is computationally infeasible for high-dimensional data, and there is no convex hull approximation in previous work, so an implementation method for approximating the IP-Delaunay graph through graph isomorphism is proposed in this patent document.

[0088] D. Examples on Large Dimensional Data

[0089] For large high-dimensional data, it is computationally infeasible to find the exact IP-Delaunay graph of the data points. Therefore, there is a need for practical and efficient graph construction and search algorithms for large-scale data in high dimensions. In this patent document, when there is a large amount of high-dimensional data, a method for constructing Implementation of a method for performing a greedy search on a graph (examples summarized in Method 4). First, an implementation of a generalized greedy search method is introduced because in one or more implementations, it will be used repeatedly during graph construction and inner product search.

[0090] 1. Greedy search implementation method

[0091] Recall that the goal of greedy search is to find x∈S to maximize f(x,q) for any query q. Here, either consider f(x,y) = -||xy|| or consider For simplicity, the nearest neighbor of x is called y when y has the maximum estimate of f(x,·). In one or more embodiments, a priority queue C can be initialized (e.g., it can be a random or carefully selected data point), and then the estimates of f(x,q) for all x∈C and all outer neighbors of these x are checked. Among those vectors estimated, C is replaced by the top k vectors in descending order of the estimate of the function f(·,q). The first k elements in C are used as the new priority queue. C can be updated until a stopping condition is reached (e.g., it no longer changes). Method 1 summarizes this process. If k=1, then this generalized greedy search is equivalent to the simple version described in Section B. This generalized greedy search allows method implementations to return an approximate top k items, which is valuable for query search and recommendation systems.

[0092] Method 1: Greedy Search (q,P,G,k,f)

[0093]

[0094]

[0095] Figure 3 A method of performing a greedy search according to an embodiment of the present disclosure is depicted. In one or more embodiments, a query element (or data point) q, a set of input data points P, a graph G (containing vertices (or elements), S and edges E), a number of candidates k to be returned, and a measure function f are given or received (305) as input. A set of priority queues C can be initialized (310) using the set of input data points P, and the elements of P are marked as checked, and the remaining vertices not in P can be marked as unchecked.

[0096] In one or more embodiments, if the number of elements in the set priority queue C is greater than the number of candidates k to be returned, C may be set (315) to the first k elements in descending order of a measure function measured relative to the query element q.

[0097] In one or more embodiments, the set of priority queues C is updated (320) to be the union of the existing set of priority queues C and an unchecked element y in the graph, element y forming a boundary in G with element x, which is in the existing set of priority queues C. The elements in the updated priority queue C may be marked (325) as checked. In one or more embodiments, if the number of elements in the set of priority queues C is greater than the number of candidates to be returned, k, then C may be set (330) to the first k elements sorted by a measure function relative to the measure of the query element q. Note that when the greedy search completes the graph construction, the measure function may be l 2 -distance, but when querying the complete graph it can be the inner product, as discussed below in Section 4 (Query Implementation).

[0098] If there are unchecked elements in the graph and the priority queue remains updated, the process returns (335) to step 320. Otherwise, in one or more embodiments, a final set of priority queues C is output (340).

[0099] 2. Graph Construction Implementation Method

[0100] The implementation of the graph construction method is shown in Method 2 below. According to Theorem 1 and Corollary 1, the best graph used is the IP-Delaunay graph on S, which is the same as the transformed l 2 -Delaunay graph subgraph isomorphism. Consider HNSW as l 2 -Delaunay graph approximation. The hierarchical structure of the Delaunay graph can be approximated by edge discrimination. In addition, directed graphs can be viewed as an approximation that reduces the total degree.

[0101] Method 2: Graph Construction (S, k, d)

[0102]

[0103] Given a dataset In one or more embodiments, Iteratively constructs a directed graph on . The directed graph can be initialized by a random graph. In each iteration, for a given graph with vertices For a directed graph G of x . x will be connected to the element closest to it, such as the candidate set C x y1 in .

[0104] Now consider the neighborhood set N(x) = {y1}. For the next nearest neighbor y, if it satisfies the boundary selection criterion: ||xy||≤min z∈N(x)||zy||, it can be added to the neighborhood set N(x). In one or more embodiments, the iterative process stops when multiple valid neighborhoods are found or when all candidate neighborhoods have been checked. Method 3, which represents an embodiment of this process, is described in the next subsection.

[0105] This boundary selection can improve the diversity of the associated boundary directions. x This step can be repeated and stopped when all elements in have been checked or the maximum out-degree d is reached. The boundary (x, y) for y∈N(x) is added to the graph. In addition, for y∈N(x), x is added to N(y). If |N(y)|>d, N(y) is updated according to the boundary selection criterion. This last step reduces the impact caused by the random order of vertices. Corollary 1 implies that the IP-Delaunay graph is l 2 -The neighborhood of 0 in the Delaunay graph (in the sense of the graph). Therefore, for any query q, a greedy search can be applied starting from the outer neighborhood of 0 (i.e., P in Method 2). The method will then search for the optimal object with respect to the inner product by greedy search. See Method 4 for an exemplary implementation.

[0106] Figure 4 An alternative method for graph construction according to embodiments of the present disclosure is depicted. In one or more embodiments, a transformed data set is formed (405) by performing a transformation on the data points (or elements) of the original data set and by adding an origin 0. The graph G is initialized (410) using at least some of the transformed data points as vertices. G can be initialized in any of a variety of ways, such as an empty graph, a random graph, or a small fully connected graph including an origin.

[0107] In one or more embodiments, a transformed data point (or element) is selected (415) that is not currently a vertex in the graph G. Then, a search such as the greedy search discussed above and l as the measure 2 -distance may be used (420) to identify a set of candidate neighborhoods. In one or more embodiments, based on l 2 The distance selects a set of nearest neighbors and adds (425) a boundary from the selected transformed point to each neighbor in the set of nearest neighbors to the graph G. In one or more embodiments, for each neighbor in the set of nearest neighbors, its set of nearest neighbors is updated (435).

[0108] In one or more embodiments, if not all transformed data elements have been processed, steps 415 to 435 are repeated (440). However, if all transformed data elements have been processed, a graph G including the transformed data points and a set of outer neighborhoods P' of the origin may be output.

[0109] In one or more embodiments, the transformed data points in the set of outer neighborhoods P' are replaced (450) with corresponding original data points to form a set of input data points P. Additionally, in one or more embodiments, the origin and its associated boundaries are removed (455) from the graph, and the transformed data points that are vertices / nodes of the graph are replaced with corresponding original data points.

[0110] Finally, if Figure 4 As shown in , a set of input data points P may be output ( 460 ) that may be used to search for a final graph G. In one or more embodiments, a set of input data points P may also be output.

[0111] 3. Neighborhood selection implementation method

[0112] Figure 5 A method for neighborhood selection according to an embodiment of the present disclosure is depicted. In one or more embodiments, an input element x, a set of k nearest neighbors C of x, and a maximum number of out-degrees d are given (505) or received as input. In one or more embodiments, a neighborhood set N of x (e.g., ) is initialized (510), which may initialize the group to an empty group. The elements in the set of k nearest neighbors C of x may be based on the l between the input element x and the elements in the set of k nearest neighbors C. 2 The distances are arranged in ascending order (515).

[0113] When the number of elements in the neighborhood group N is less than or equal to the maximum number of out-degrees d, and the number of iterations is less than or equal to the number of elements in the group of k nearest neighbors C, the following steps are performed (520). First, if for iteration y i The l between the input element x and the elements in the set of k nearest neighbors C 2 The distance is less than or equal to y for this iteration i The minimum l between each element in the neighborhood group N and the elements in the group of k nearest neighbors C 2 distance, then y i Add (525) to N. Select the next element from the set of k nearest neighbors C, and repeat this process until the while condition 520 indicates to stop. In one or more embodiments, the next element can be selected by using an iteration index i that is incremented in each iteration.

[0114] Once the while process ends, the final neighborhood group N may be output (530) and returned.

[0115] Method 3 proposes an alternative implementation for selecting a neighborhood according to an embodiment of the present disclosure.

[0116] Method 3: Select neighborhood (x,C,d)

[0117]

[0118]

[0119] 4. Query implementation method

[0120] Figure 6 An overall method according to an embodiment of the present disclosure is depicted. In one or more embodiments, a set of queries Q, a data set S, a number of elements to be returned K, a size k of a candidate set for graph construction, a size l for greedy search, and a maximum out-degree d of the graph are given (605) or received as input. In one or more embodiments, a graph is constructed (610) using the data set S, the size k of the candidate set for graph construction, and the maximum out-degree d of the graph. The above presents an exemplary embodiment of graph construction.

[0121] It should be noted that if the graph has already been constructed, steps 605 and 610 may be omitted.

[0122] Given the final graph, in one or more embodiments, a search (e.g., a greedy search) is performed (615) using the graph and using the inner product as a ranking metric for query elements in a set of one or more queries. After the search, a set of top K elements C is output (620) for the query elements in the set of queries. q In one or more embodiments, the output may be presented in descending order of the inner product with the query element q.

[0123] Method 4 depicts an alternative implementation of the overall method according to an embodiment of the present disclosure.

[0124] Method 4: MIPS(Q,S,K,k,l,d)

[0125] 1: Input: a set of queries Q, a dataset S, the number of elements to be returned K, the size k of the candidate set for graph construction and the size l for greedy search, and the maximum out-degree d of the graph.

[0126] 2: (G, P) ← graph construction (S, k, d).

[0127] 3: for q∈Q do

[0128] 4: C q ← Greedy search (q, P, G, l, inner product)

[0129] 5: Output: For q∈Q, a set of the first K elements sorted in descending order of their inner product with q

[0130] E. Experiment

[0131] It should be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using a specific embodiment or embodiments; therefore, neither these experiments nor their results should be used to limit the scope of the disclosure of the current patent document.

[0132] In this section, the implementation is compared with the state-of-the-art MIPS methods on four datasets (see Table 1): dataset A, dataset B, dataset C, and dataset D. For dataset A, 50-dimensional vectors are used. For datasets B and C, matrix factorization methods are used to obtain 100-dimensional latent vectors.

[0133] Table 1: Statistics of the dataset.

[0134] Dataset #Basic data #Query data #Dimension #extremum %extremum Dataset A ～17800 1000 50 8017 45.12％ Dataset B ～104700 7748 100 3169 3.03％ Dataset C ～25800 25677 100 722 2.80％ Dataset D ～1000000 1000 100 304431 30.44％

[0135] The ground truth for each query vector is the previous, top ten, and top hundred measured by the inner product. Only a small fraction of the data points can be the previous solution of Equation 1, i.e., the extreme points in Remark 1, and their percentage is an important feature of the dataset in the MIPS problem. The percentage of extreme points for each dataset is estimated as follows: For each vector x in the base case, calculate its inner product with all vectors y in the base case (including x itself). We then count the number of unique previous vectors y (i.e., extreme points) and calculate the percentage of extreme points by dividing by the size of the dataset (i.e., the last column of Table 1). This may not be an exact estimate, but it is a tight lower bound.

[0136] 1. Experimental Setup

[0137] For convenience, the embodiments of the methods disclosed herein may generally be referred to as Figure, and will The implementation of the graph is compared with three previous state-of-the-art MIPS methods (Greedy-MIPS, ip-NSW, and Range-LSH), which are the most representative of MIPS. In Range-LSH, the dataset is first divided into small subsets according to the l2 norm sorting, and then the data is normalized using the local maximum l2 norm in each subset. This overcomes the performance limitation due to the long-tail distribution of the data norm. The authors of the Greedy-MIPS method used the upper bound of the inner product as an approximation of MIPS and designed a greedy search algorithm to find this approximation, called Greedy-MIPS. Their original implementation was used. The open source code of ip-NSW uses HNSW instead of NSW for graph construction. It was found that the hierarchical structure and heuristic boundary selection in HNSW did not significantly improve the performance of ip-NSW; see Figure 7 In order to provide a comprehensive evaluation, HNSW and SONG were used to achieve Fig. Implementation. All compared methods have adjustable parameters. To get a fair comparison, all parameters are varied on a fine grid.

[0138] As evaluation measures, the trade-off between calls and queries per second (QPS) and calls and compute percentage was chosen. The calls and queries per second relationship reports the number of queries that the method can handle per second at each call level. Ideally, if a higher call level is desired, the method can handle as many queries as possible (i.e., more efficient). The calls and compute percentage relationship examines the pairwise computations that are as few as possible at each call level. For each method, multiple points scattered on the plane were obtained by adjusting the parameters. To plot the curve, first find the best result max along the x-axis (i.e., calls) x Then, by evenly splitting from 0 to max x 100 buckets are generated for a range of . For each bucket, the best result (i.e. the maximum number of queries per second) is selected along the y-axis. If there is no data point in the bucket, it will be ignored. In this way, there are at most 100 pairs of data for drawing the curve. All experiments were performed on a 2X 3.00GHz 8-core i7-5960XCPU server with 32GB of memory.

[0139] 2. Experimental results

[0140] FIG. 8A to FIG. 8C The experimental results of the relationship between calls and queries per second (QPS) are shown in Figure 1. Each column corresponds to a data set, and the graphs in each row are the results for the top one, top ten, and top one hundred tags, respectively. It can be seen that Graph implementations outperform previous state-of-the-art methods in most cases on all datasets.

[0141] An interesting fact is the influence of the percentage of extreme points across different datasets. The implementation scheme is inspired by the typical phenomenon that a dataset has a low percentage of extreme points. Therefore, the constructed approximate Delaunay graph will be effective for maximum inner product retrieval. However, it can be seen that the proposed implementation performs very well for datasets with a high percentage of extreme points, such as dataset A with more than 45% extreme points and dataset D with more than 30% extreme points. Results for different ground truth label settings are shown, which show that the tested implementation performs well in various cases not only for the first label but also for the first ten and first hundred labels. These results demonstrate the high performance of the proposed implementation in MIPS. FIG. 4 shows the robustness of the implementation scheme.

[0142] On the contrary, it is difficult to distinguish which baseline performs better than others on all datasets. Range-LSH performs relatively well on dataset A, but is much worse than other methods on the other three datasets. The baseline ip-NSW performs well on datasets with a high percentage of extreme points (e.g., datasets A and D), but becomes very bad on other datasets. Greedy-MIPS outperforms ip-NSW on datasets with a low percentage of extreme points (e.g., datasets B and C) at some call levels.

[0143] The result of calling and calculating the percentage relationship is Fig. 9 Due to limited space, only the first 10 results are shown. The first and first 100 results can be found in the appendix. It should be noted that this measurement may not be meaningful for Greedy-MIPS. The results of the relationship between call and calculation percentage are shown in Fig. 9 In this view, The implementation in Figure 1 performs best in all cases. In this view, Range-LSH performs comparable to the others on smaller datasets (i.e., the first three). The call and computation percentage relationship does not take into account the cost of different index structures. FIG. 8A to FIG. 8C As shown in , although Range-LSH performs well in this view, its total time cost is much higher than the others. The possible reason is that the table-based index used in Range-LSH is not effective in search. In addition, Range-LSH also performs poorly on the much larger dataset D. The curve of Range-LSH cannot be displayed within the range of dataset D.

[0144] In addition, through ip-NSW and The time cost of graph construction in the graph implementation method is shown in Table 2. It can be seen that The graph implementation method consumes 13.7% to 65.5% less time in index construction than ip-NSW, which brings great benefits to practical applications. The reason is that the search based on metric measurement (i.e., l2) (in graph construction) is more efficient than the search based on inner product.

[0145] Table 2: Graph construction time in seconds

[0146]

[0147]

[0148] 3. Implementation through SONG

[0149] In order to eliminate the deviation in the embodiment, another SONG implementation method of searching on the graph platform is implemented. The results are shown in the following figure: Fig.10 As shown in . It can be seen that for In both the ip-NSW and SONG implementations, the SONG implementation is more efficient than the HNSW implementation, but their priority order remains the same. The implementation schemes all perform better than ip-NSW.

[0150] F. Some conclusions

[0151] Maximum inner product search (MIPS) is a challenging problem with wide applications in search and machine learning. In this patent document, a novel implementation of a MIPS graph search method is proposed. From the perspective of computational geometry, it is shown that kind Under the transformation, the Delaunay graph of the inner product is 2 -norms are isomorphic. Based on this observation, we propose to convert l 2 -Delaunay graph into an IP-Delaunay graph. MIPS via greedy search can then be performed on the transformed graph. This demonstrates that this embodiment provides an effective and efficient solution for MIPS.

[0152] The embodiments herein focus on fast search under a non-metric measure, inner product. In addition to inner product, more complex measures such as Bregman divergence, maximum kernel, and more general measures are also studied. It should be noted that the embodiments herein can be extended to these measures. In addition, a GPU-based system for fast ANN search and MIPS can also be used in conjunction with the embodiments or their adaptations, which has been shown to be very effective for general ANN tasks.

[0153] G. Appendix

[0154] 1. Search on the Delaunay graph

[0155] Greedy search on the Delaunay graph is necessary and sufficient for ANN to search for the global optimum. We generalize this sufficiency to a larger class f. First, consider the general optimization problem. Let We consider the dataset And for q∈X, the goal is to solve the optimization problem

[0156]

[0157] Assuming f is continuous, we obtain the following theorem.

[0158] Theorem 2: For a given f, we assume that for any dataset S, each Voronoi cell l R i are all connected. Let G = (S, E) be the Delaunay graph of the Voronoi cell. Then for any q∈X, a simple greedy search on the Delaunay graph returns the solution of formula (4). In other words, let N(x i )={x j ∈S:{x i ,x j}∈G} is x i Neighborhood on the Delaunay graph. If x i satisfy:

[0159]

[0160] Then x i is the solution of formula (4). In contrast, for any G′ that does not contain a Delaunay graph as a subgraph, there exists a query q∈Y such that a greedy search on G′ does not always retrieve all global maxima.

[0161] By the assumption on f, we have

[0162]

[0163] is connected and Therefore, we can limit the path So that c(0)∈R i and c(1) = q. For every x j ∈S, f(x j ,f(x j ,c(0))≤f(x i ,c(0)). If f(x j ,c(1))≥f(x i ,c(1)), then by the intermediate value theorem, there exists t∈[0,1] such that f(x j ,c(t))=f(x i ,c(t)). Therefore and x j For x i neighborhood on G. In this case, by formula (5), we have f(x i ,q)=f(x j ,q). Therefore, for x j ∈S, we have f(x i ,q)≥f(x j ,q.

[0164] On the contrary, if G' misses the boundary in the Delaunay graph, then there exists xi and x j Make R i ∩ but Assume query q∈R i ∩R j And the initial point is x i , then x i and x j Both are global maxima of f(·,q). j Not x i Neighborhood of x i is the global maximum, so the greedy search stops at this step. Therefore, x j cannot be found to be a global maximum.

[0165] 2. Supplementary explanation on hypothesis 1

[0166] Assumption 1 simplifies the argument in part c. To better understand Assumption 1, we develop the following equivalent expression.

[0167] Proposition 3: The following are equivalent:

[0168] S satisfies equation (A1).

[0169] The convex hull of S contains 0 as an interior point.

[0170] For each There exists x∈S such that

[0171] Assuming 0 is not an interior point of Conv(S), there exists a closed half-space H containing S with boundary point 0. H is a convex cone, so

[0172] For each There exists β>0 such that βa∈Conv(S).

[0173] So for some α i ≥0, but So there exists x∈S such that

[0174] Assumptions Then for some closed half-space H, for So that it is perpendicular to the boundary line H, there is no x∈S such that

[0175] Assuming that Assumption 1 is not satisfied, the MIPS problem is still interesting. This situation will be discussed in the following two cases.

[0176] Case 1. If Assumption 1 is not true, but the query is always located in the conical hull of the dataset S, then the method is still valid because for each query, kind After the transformation, the correct solution to the MIPS problem is still in the neighborhood of 0.

[0177] Case 2. Assuming Assumption 1 is not true and the query can be any point in Euclidean space, this method may not work. However, the graph construction method can be slightly changed as follows. We find the center of the dataset, say c, and then we apply the transformation to each data point:

[0178]

[0179] To obtain (Although the denominator may be ||x|| n , n>1). It should be noted that since it has the form of formula (3), such g(x) is still kind Transformation. It is not difficult to check that the IP-Delaunay diagram introduced in Corollary 1 and l 2 -Isomorphisms between subgraphs of the Delaunay graph. However, in this special case, this approach is recommended. Centering on the data point changes all norms, and the length of the vector determines the likelihood of return in the MIPS problem.

[0180] 3. Proof of Theorem 1

[0181] By Definition 1, the inner product and x i The Voronoi cell R i for

[0182] for k∈[n]}.

[0183] Similarly,

[0184] for k∈[n]}.

[0185] By Definition 2, (a) means that there exists a∈R i ∩R j . a also satisfies

[0186] If a satisfies statement (b), then a∈R i∩R j , which implies (a) by Definition 2.

[0187] First, note that x i =y i / ||y i || 2 , then suppose as well as Note from Proposition 3(c) that b>0. Then, we have

[0188]

[0189] therefore, Using formula (A1), we have So for x∈S and y=x / ||x|| 2 ∈S′,

[0190]

[0191] therefore, Since this is true for all x∈S, we have ||c||≤min y∈S′ ||yc||. Because We can j Repeat the argument to obtain statement (c).

[0192] This can be proved by observing Each step is reversible to prove.

[0193] This is due to the empty sphere criterion. See Proposition 2.

[0194] 4. Additional empirical experiments

[0195] In the paper by Morozov and Babenko (mentioned above), it is claimed that their algorithm can be used with any graph construction algorithm, including NSW and HNSW. Fig. 4. Implementation of the proposed method compared with different versions of ip-NSW. Boundary selection is a contribution of HNSW. However, its applicability on non-metric measurements is not guaranteed. We compared the MIPS efficiency of ip-NSW with and without the boundary selection step and found interesting observations. FIG. 11A to FIG. 11C It shows that the performance of boundary selection is poor for datasets B and C, and the effect is not obvious on the other two datasets. When the proportion of extreme points is small, boundary selection may not help.

[0196] The hierarchical graph structure in HNSW is used to perform multi-scale hopping. The implementation diagram can help you find a good starting point, so let's see if ip-NSW can be used. It would be interesting to find a starting point for the graph implementation to work effectively. Here we design a variant of ip-NSW, ip-NSW-init, which abandons the hierarchical index structure but adopts The graph implementation finds the starting point. For each query, we perform a greedy search using This step is done offline and the time cost will not be included in the time cost of ip-NSW-init. The result is as follows Figures 12A to 12C If ip-NSW is from Starting the search from the starting point found by the graph implementation significantly improves the performance in the previous inner product search. However, this difference disappears if we consider the first 100 results. We also compare the impact of the hierarchical graph structure on the performance of ip-NSW. FIG. 13A to FIG. 13C As can be seen, its impact is very small.

[0197] Figures 14A to 14B Completed Part E Fig. 9 The experimental results. FIG. 14A to FIG. 14B Experimental results of the call versus compute percentage relationship according to an embodiment of the present disclosure are depicted. The remaining results for the first and first one hundred tags are shown. The curve for Range-LSH on dataset D is not within the display range.

[0198] H. Computing System Implementation

[0199] In one or more embodiments, aspects of the present patent document may relate to, may include, or may be implemented on one or more information processing systems (or computing systems). An information processing system / computing system may include any means or combination of means operable to calculate, compute, determine, classify, process, transmit, receive, retrieve, initiate, route, exchange, store, display, communicate, visualize, detect, record, reproduce, process, or utilize any form of information, intelligence, or data. For example, a computing system may be or may include a personal computer (e.g., a laptop), a tablet, a mobile device (a personal digital assistant (PDA), a smartphone, a tablet phone, a tablet, etc.), a smart watch, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and may vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources (such as a central processing unit (CPU) or hardware or software control logic), read-only memory ROM, and / or other types of memory. Additional components of the computing system may include one or more disk drives, one or more network ports for communicating with external devices, and various input and output (I / O) devices (e.g., keyboard, mouse, stylus, touch screen, and / or video display). The computing system may also include one or more buses operable to transmit communications between the various hardware components.

[0200] Fig.15 A simplified block diagram of an information processing system (or computing system) according to an embodiment of the present disclosure is depicted. Although it should be understood that the computing system may be configured differently and include different components, including Fig.15 Although fewer or more components may be shown in , it should be understood that the functionality shown for system 1500 is operable to support various embodiments of computing systems.

[0201] like Fig.15 As shown, computing system 1500 includes one or more central processing units (CPUs) 1501, which provide computing resources and control the computer. CPU 1501 may be implemented with a microprocessor, etc., and may also include one or more graphics processing units (GPUs) 1502 and / or floating-point coprocessors for mathematical calculations. In one or more embodiments, one or more GPUs 1502 may be incorporated into a display controller 1509, such as part of one or more graphics cards. System 1500 may also include system memory 1519, which may include RAM, ROM, or both.

[0202] like Fig.15As shown, multiple controllers and peripherals may also be provided. Input controller 1503 represents an interface to various input devices 1504, such as a keyboard, a mouse, a touch screen, and / or a stylus. Computing system 1500 may also include a storage controller 1507, which is used to interface with one or more storage devices 1508, each of which includes a storage medium (such as a tape or disk) or an optical medium (which can be used to record a program of instructions for operating systems, utilities, and applications, which may include implementations of programs that implement various aspects of the present disclosure). Storage device 1508 may also be used to store processed data or data to be processed according to the present disclosure. System 1500 may also include a display controller 1509, which is used to provide an interface for a display device 1511, which may be a cathode ray tube (CRT) display, a thin film transistor (TFT) display, an organic light emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. The computing system 1500 may also include one or more peripheral device controllers or interfaces 1505 for one or more peripheral devices 1506. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, etc. The communication controller 1514 may interface with the one or more communication devices 1502, which enables the system 1500 to connect to remote devices through any of a variety of networks, including the Internet, cloud resources (e.g., Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) cloud, etc.), local area network (LAN), wide area network (WAN), storage area network (SAN), or through any suitable electromagnetic carrier signal (including infrared signal). As depicted in the embodiment, the computing system 1500 includes one or more fans or fan trays 1518 and one or more cooling subsystem controllers 1517, which monitor the thermal temperature of the system 1500 (or its components) and operate the fans / fan trays 1518 to help regulate the temperature.

[0203] In the system shown, all major system components may be connected to bus 1516, which may represent more than one physical bus. However, various system components may or may not be physically close to each other. For example, input data and / or output data may be remotely transmitted from one physical location to another physical location. In addition, programs that implement various aspects of the present disclosure may be accessed from a remote location (e.g., a server) via a network. Such data and / or programs may be transmitted via any of a variety of machine-readable media, including, for example: magnetic media such as hard disks, floppy disks, and tapes; optical media such as compact disks (CDs) and holographic devices; magneto-optical media; and hardware devices that are specifically configured to store or store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (e.g., 3D XPoint-based devices), and ROM and RAM devices.

[0204] Aspects of the present disclosure may utilize instructions for one or more processors or processing units to perform steps encoded on one or more non-transitory computer-readable media. It should be noted that one or more non-transitory computer-readable media should include volatile memory and non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. The functions implemented by hardware can be implemented using ASICs, programmable arrays, digital signal processing circuits, etc. Therefore, the term "means" in any claim is intended to cover both software implementations and hardware implementations. Similarly, the term "computer-readable medium or medium" as used herein includes software and / or hardware or a combination thereof having an instruction program implemented thereon. With these alternative implementations contemplated, it should be understood that the accompanying drawings and accompanying descriptions provide those skilled in the art with functional information required to write program code (i.e., software) and / or manufacture circuits (i.e., hardware) to perform the desired processing.

[0205] It should be noted that embodiments of the present disclosure may also relate to computer products having non-transitory tangible computer-readable media having computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be known or available to those skilled in the relevant art. Examples of tangible computer-readable media include, for example: magnetic media such as hard disks, floppy disks, and tapes; optical media such as CDs and holographic devices; magneto-optical media; and hardware devices specially configured to store or store and execute program code, such as ASICs, programmable logic devices (PLDs), flash memory devices, other non-volatile memory (NVM) devices (e.g., 3D XPoint-based devices), and ROM and RAM devices. Examples of computer code include machine code (e.g., code produced by a compiler) and files containing higher-level code that can be executed by a computer using an interpreter. Embodiments of the present disclosure may be implemented in whole or in part as machine-executable instructions in program modules that can be executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located in local, remote, or both local and remote locations.

[0206] Those skilled in the art will recognize that the computing system or programming language is not important to the practice of the present disclosure. Those skilled in the art will also recognize that a plurality of the above elements can be physically and / or functionally divided into modules and / or sub-modules or combined together.

[0207] Those skilled in the art will appreciate that the foregoing examples and implementations are exemplary and do not limit the scope of the present disclosure. It is intended to illustrate that all permutations, enhancements, equivalents, combinations or improvements of the present disclosure that would be apparent to a person skilled in the art after reading this specification and studying the drawings are included within the true spirit and scope of the present disclosure. It should also be noted that the elements of any claim may be arranged differently, including having multiple dependencies, configurations and combinations.

Claims

1. A computer-implemented method comprising: forming a transformed data set by transforming a data set including original data elements and by adding the original elements to the transformed data set, wherein the data set includes data applied to at least one of a recommender system, natural language processing and a multi-class or multi-label classifier, computational advertising for a search engine, the data set being connected with a graph-based index for a maximum inner product search and an approximate nearest neighbor search; Use l 2 distance constructing an approximation graph of the transformed data set, wherein at least some of the data elements of the transformed data set are vertices in the approximation graph, and connections between vertices in the approximation graph are edges; identifying a set of neighbors of the original element on the approximation graph; removing the original element and its boundaries from the approximation graph; obtaining a final graph by replacing the vertices in the approximate graph with corresponding original data elements, the approximate graph comprising data elements from the transformed data set; outputting the final graph for searching using the inner product as the ranking metric, and Given a query data element, a search is performed on the final graph using inner product as the ordering measure to identify a set of data elements for the query data element.

2. The computer-implemented method of claim 1, wherein: The transformation is Transformation.

3. The computer-implemented method of claim 1, wherein: The search is a greedy search.

4. The computer-implemented method of claim 1, wherein: Use l 2 The step of constructing an approximate graph of the transformed data set by distance, wherein at least some of the data elements of the transformed data set are vertices in the approximate graph, and connections between vertices in the approximate graph are edges, comprises: initializing the approximation graph using at least some of the transformed data elements as vertices; While all of the transformed data elements have not yet been processed, performing the steps comprising: selecting a transformed data element that is not currently a vertex in the approximation graph; Using greedy search and l 2 Distance is used as a measure to identify a set of candidate neighbors; Based on l 2 distance selecting a set of nearest neighbors from the set of candidate neighbors; adding a boundary to the approximation graph from the selected transformed data element to each neighbor in the set of nearest neighbors; and For each neighbor in the set of nearest neighbors, updating its set of nearest neighbors; and The output comprises the approximation graph of the transformed data elements.

5. The computer-implemented method of claim 4, further comprising: outputting a set of neighborhoods of the original element comprising the transformed data element; replacing the transformed data elements in the set of neighborhoods with corresponding original data elements to form a set of input data elements; as well as Output the set of input data elements.

6. The computer-implemented method of claim 5, further comprising: Given a query data element, a set of input data elements, the final graph including vertices and edges, and the number of candidates k to be returned: Initializing a set of priority queues with the set of input data elements; marking the vertices of the final graph that are members of the set of input data elements as checked, and marking the remaining vertices of the final graph as unchecked; In response to the number of data elements in the set of priority queues being greater than the number of candidates to be returned, setting the set of priority queues to the first k data elements in descending order of inner product, the inner product being measured relative to the query data element; When there are unchecked data elements in the final graph and the set of priority queues remain updated, the following steps are performed: updating the set of priority queues to be a union of the existing set of priority queues and unchecked data elements in the final graph, the unchecked data elements forming a boundary in the final graph, the final graph having the data elements in the existing set of priority queues; marking the updated data elements in the set of priority queues as checked; as well as In response to the number of data elements in the set of priority queues being greater than the number of candidates to be returned, setting the set of priority queues to the first k data elements in descending order of inner product, the inner product being measured relative to the query data element; and Output the final set of priority queues.

7. A non-transitory computer readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, cause the performance of operations comprising: A transformed data set is formed by transforming a data set including the original data elements and by adding the original elements to the transformed data set, wherein: The dataset includes data applied to at least one of a recommendation system, natural language processing and a multi-class or multi-label classifier, and computational advertising for a search engine, and the dataset is connected with a graph-based index for a maximum inner product search and an approximate nearest neighbor search; Use l 2 distance constructing an approximation graph of the transformed data set, wherein at least some of the data elements of the transformed data set are vertices in the approximation graph, and connections between vertices in the approximation graph are edges; identifying a set of neighbors of the original element on the approximation graph; removing the original element and its boundaries from the approximation graph; obtaining a final graph by replacing the vertices in the approximate graph with corresponding original data elements, the approximate graph comprising data elements from the transformed data set; outputting the final graph for searching using the inner product as the ranking metric, and Given a query data element, a search is performed on the final graph using inner product as the ordering measure to identify a set of data elements for the query data element.

8. The non-transitory computer readable medium or media of claim 7, wherein: The transformation is Transformation.

9. The non-transitory computer readable medium or media of claim 7, wherein: The search is a greedy search.

10. The non-transitory computer readable medium or media of claim 7, wherein: Use l 2 The step of constructing an approximate graph of the transformed data set by distance, wherein at least some of the data elements of the transformed data set are vertices in the approximate graph, and connections between vertices in the approximate graph are edges, comprises: initializing the approximation graph using at least some of the transformed data elements as vertices; While all of the transformed data elements have not yet been processed, performing the steps comprising: selecting a transformed data element that is not currently a vertex in the approximation graph; Using greedy search and l 2 Distance is used as a measure to identify a set of candidate neighbors; Based on l 2 distance selecting a set of nearest neighbors from the set of candidate neighbors; adding a boundary to the approximation graph from the selected transformed data element to each neighbor in the set of nearest neighbors; and For each neighbor in the set of nearest neighbors, updating its set of nearest neighbors; and The output comprises the approximation graph of the transformed data elements.

11. The non-transitory computer readable medium or media of claim 10, the operations further comprising the steps of: outputting a set of neighborhoods of the original element comprising the transformed data element; replacing the transformed data elements in the set of neighborhoods with corresponding original data elements to form a set of input data elements; as well as Output the set of input data elements.

12. The non-transitory computer readable medium or media of claim 11, the operations further comprising the steps of: Given a query data element, a set of input data elements, the final graph including vertices and edges, and the number of candidates k to be returned: Initializing a set of priority queues with the set of input data elements; marking the vertices of the final graph that are members of the set of input data elements as checked, and marking the remaining vertices of the final graph as unchecked; In response to the number of data elements in the set of priority queues being greater than the number of candidates to be returned, setting the set of priority queues to the first k data elements in descending order of inner product, the inner product being measured relative to the query data element; When there are unchecked data elements in the final graph and the set of priority queues remain updated, the following steps are performed: updating the set of priority queues to be the union of the existing set of priority queues and the unchecked data elements in the final graph, The unchecked data elements form boundaries in the final graph, The final graph has data elements in the existing set of priority queues; marking the updated data elements in the set of priority queues as checked; as well as In response to the number of data elements in the set of priority queues being greater than the number of candidates to be returned, setting the set of priority queues to the first k data elements in descending order of inner product, the inner product being measured relative to the query data element; and Output the final set of priority queues.

13. A system comprising: one or more processors; as well as A non-transitory computer readable medium or media comprising one or more sequences of instructions which, when executed by at least one of the one or more processors, cause operations to be performed comprising: forming a transformed data set by transforming a data set including original data elements and by adding the original elements to the transformed data set, wherein the data set includes data applied to at least one of a recommender system, natural language processing and a multi-class or multi-label classifier, computational advertising for a search engine, the data set being connected with a graph-based index for a maximum inner product search and an approximate nearest neighbor search; Use l 2 distance constructing an approximation graph of the transformed data set, wherein at least some of the data elements of the transformed data set are vertices in the approximation graph, and connections between vertices in the approximation graph are edges; identifying a set of neighbors of the original element on the approximation graph; removing the original element and its boundaries from the approximation graph; obtaining a final graph by replacing the vertices in the approximate graph with corresponding original data elements, the approximate graph comprising data elements from the transformed data set; outputting the final graph for searching using the inner product as the ranking metric, and Given a query data element, a search is performed on the final graph using inner product as the ordering measure to identify a set of data elements for the query data element.

14. The system of claim 13, wherein: The search is a greedy search.

15. The system of claim 13, wherein: Use l 2 The step of constructing an approximate graph of the transformed data set by distance, wherein at least some of the data elements of the transformed data set are vertices in the approximate graph, and connections between vertices in the approximate graph are edges, comprises: initializing the approximation graph using at least some of the transformed data elements as vertices; While all of the transformed data elements have not yet been processed, performing the steps comprising: selecting a transformed data element that is not currently a vertex in the approximation graph; Using greedy search and l 2 Distance is used as a measure to identify a set of candidate neighbors; Based on l 2 distance selecting a set of nearest neighbors from the set of candidate neighbors; adding a boundary to the approximation graph from the selected transformed data element to each neighbor in the set of nearest neighbors; and For each neighbor in the set of nearest neighbors, updating its set of nearest neighbors; and The output comprises the approximation graph of the transformed data elements.

16. The system of claim 15, wherein: The operation also includes the following steps: outputting a set of neighborhoods of the original element comprising the transformed data element; replacing the transformed data elements in the set of neighborhoods with corresponding original data elements to form a set of input data elements; as well as Output the set of input data elements.

17. In the system of claim 16, wherein: The operation also includes the following steps: Given a query data element, a set of input data elements, the final graph including vertices and edges, and the number of candidates k to be returned: Initializing a set of priority queues with the set of input data elements; marking the vertices of the final graph that are members of the set of input data elements as checked, and marking the remaining vertices of the final graph as unchecked; In response to the number of data elements in the set of priority queues being greater than the number of candidates to be returned, setting the set of priority queues to the first k data elements in descending order of inner product, the inner product being measured relative to the query data element; When there are unchecked data elements in the final graph and the set of priority queues remain updated, the following steps are performed: updating the set of priority queues to be a union of the existing set of priority queues and unchecked data elements in the final graph, the unchecked data elements forming a boundary in the final graph, the final graph having the data elements in the existing set of priority queues; marking the updated data elements in the set of priority queues as checked; as well as In response to the number of data elements in the set of priority queues being greater than the number of candidates to be returned, setting the set of priority queues to the first k data elements in descending order of inner product, the inner product being measured relative to the query data element; and Output the final set of priority queues.

Images