Systems and methods for map-reduce vector indexing and search

Abstract

A computer-implemented method including forming, based on a similarity between a plurality of vectors, a plurality of clusters, where each of the plurality of vectors is representative of a respective one of a plurality of data objects in a dataset, and where each cluster of the plurality of clusters comprises a different subset of the plurality of vectors; calculating a centroid for an individual cluster of the plurality of clusters based on a respective subset of the plurality of vectors in the cluster; indexing the respective subset of the plurality of vectors in the individual cluster based on an approximate nearest neighbor (ANN) search to generate an individual hierarchical graph-based indexing database; storing, in a database, the centroid of the individual cluster in association with the corresponding individual hierarchical graph-based indexing database; receiving a query associated with the dataset; and responding to the query based on the database.

Description

Systems and Methods for Map-Reduce Vector Indexing and SearchTECHNICAL FIELD

[0001] The present disclosure is generally related to vector databases and machine learning systems and, in particular embodiments, to map-reduce vector indexing and search.BACKGROUND

[0002] Recent advancements in machine learning (ML) technologies (e.g., large-language models (LLMs)) enable a wide range of applications, such as allowing computers to understand, translate, predict, and generate text and other forms of content based on knowledge gained from a massive amount of data. LLMs can be applied to numerous fields in the industry. As an example, retailers and service providers can use LLMs to provide improved customer experience through chatbots, artificial intelligence (Al) assistants, etc. As another example, businesses can train LLMs based on historical customer feedback and activities to provide product and / or service recommendations. As yet another example, search engines can use LLMs to provide more direct, human-like answers. As a further example, LLMs are increasingly being explored to act as tutors, offering personalized learning experiences by providing interactive dialogues, real-time feedback, assessing individual progress, and adapting to individual learning needs.SUMMARY

[0003] The disclosed aspects / embodiments of the present disclosure provide techniques, methods and devices or systems for performing efficient vector indexing and search (e.g., for LLM processing and / or any Al model processing). More specifically, a map-reduce approach is used to reduce the complexity for building and searching a vector database by clustering vectors (e.g., embedding vectors) into multiple clusters (e.g., N clusters, where N is an integer value greater than 1) prior to applying a hierarchical navigational small world (HNSW) algorithm (e.g., an approximate nearest neighbor (ANN) algorithm) to each cluster independently. Stated differently, the map-reduce approach may include clustering vectors to centroids and sharding (e.g., mapping or associating) the centroids to HNSW index shards (e.g., HNSW index databases). A centroid may refer to a geo-centroid (a center point) of a cluster in a multi-dimensional vector space. During a search for a query, the map-reduce approach may locate N1 most relevant cluster(s) (e.g., closestto the query) and may only search the HNSW index shard(s) corresponding to those N1 most relevant cluster(s). The map-reduce approach may retrieve K nearest-neighbors from each HNSW index shard and reduce the N1 sets of K nearest-neighbors to K nearest-neighbors.

[0004] A first aspect of the embodiments of the present disclosure relates to a computer- implemented method comprising forming, based on a similarity between a plurality of vectors, a plurality of clusters, wherein each of the plurality of vectors is representative of a respective one of a plurality of data objects in a dataset, and wherein each cluster of the plurality of clusters comprises a different subset of the plurality of vectors; calculating a centroid for an individual cluster of the plurality of clusters based on a respective subset of the plurality of vectors in the cluster; indexing the respective subset of the plurality of vectors in the individual cluster based on an approximate nearest neighbor (ANN) search to generate an individual hierarchical graph-based indexing database; storing, in a database, the centroid of the individual cluster in association with the corresponding individual hierarchical graph-based indexing database; receiving a query associated with the dataset; and responding to the query based on the database.

[0005] Optionally, in any of the preceding aspects, another implementation of the aspect provides that a first vector of the plurality of vectors comprises one or more embeddings associated with one or more attributes of a respective one of the plurality of data objects.

[0006] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the one or more embeddings in the first vector are based on a semantic analysis of the respective one of the plurality of data objects.

[0007] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the calculating the centroid for the individual cluster is further based on an average of the respective subset of the plurality of vectors within the individual cluster.

[0008] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the individual hierarchical graph-based indexing database for the individual cluster comprises a plurality of layers comprising a plurality of nodes corresponding to the respective subset of the plurality of vectors, and wherein a first layer of the plurality of layers comprises a fewer number of nodes connected by one or more longer distance connections than a second layer of the plurality of layers.

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the ANN search used for indexing the respective subset of the plurality of vectors inthe individual cluster to generate the individual hierarchical graph-based indexing database is based on a hierarchical navigational small world (HNSW) algorithm.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the indexing the respective subset of the plurality of vectors in the individual cluster based on the ANN search comprises generating a first hierarchical graph-based indexing database for a first cluster of the plurality of clusters concurrent with generating a second hierarchical graphbased indexing database for a second cluster of the plurality of clusters.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the method further comprises receiving a subsequent data object; calculating a subsequent vector representative of the subsequent data object; and inserting the subsequent vector into a first cluster of the plurality of clusters based on a distance between the subsequent vector and a respective centroid of the first cluster satisfying a threshold, wherein the inserting comprises updating a respective individual hierarchical graph-based indexing database corresponding to the first cluster.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the inserting the subsequent vector into the first cluster is further based on a comparison of a first distance between the subsequent vector and a first nearest-neighbor in the first cluster and a second distance between the subsequent vector and a second nearest-neighbor in a second cluster of the plurality of clusters.

[0013] A second aspect of the embodiments of the present disclosure relates to a computer- implemented method comprising receiving a query associated with a dataset; calculating a query vector based on one or more attributes of the query; selecting one or more centroids from a plurality of centroids of respective ones of a plurality of clusters of vectors, wherein the selecting is based on a distance between the query vector and each of the one or more centroids, and wherein each vector of the plurality of clusters of vectors is representative of a respective one of a plurality of data objects in the dataset; performing, based on the query vector, an approximate nearest neighbor (ANN) search in each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids to identify, from a respective one of the plurality of clusters of vectors, one or more nearest vectors to the query vector; and generating, based on the identified one or more nearest vectors, a response to the query.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides that a first vector of the plurality of clusters of vectors comprises embeddings associated with one or more attributes of a respective one of the plurality of data objects.

[0015] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the one or more embeddings in the first vector are based on a semantic analysis of the respective one of the plurality of data objects.

[0016] Optionally, in any of the preceding aspects, another implementation of the aspect provides that a first centroid of the plurality of centroids is an average of a respective cluster of the plurality of clusters of vectors.

[0017] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the selecting is further based on the distance between the query vector and each of the one or more centroids satisfying a threshold.

[0018] Optionally, in any of the preceding aspects, another implementation of the aspect provides that each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids comprises a plurality of layers comprising a plurality of nodes corresponding to a respective cluster of vectors, and wherein a first layer of the plurality of layers comprises a fewer number of nodes connected by one or more longer distance connections than a second layer of the plurality of layers.

[0019] Optionally, in any of the preceding aspects, another implementation of the aspect provides that each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids is based on a hierarchical navigational small world (HNSW) algorithm.

[0020] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the performing the ANN search in each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids comprises performing, based on the query vector, a first ANN search in a first individual hierarchical graph-based indexing database associated with a first centroid of the one or more selected centroids to identify one or more first nearest vectors; performing, based on the query vector, a second ANN search in a second individual hierarchical graph-based indexing database associated with a second centroid of the one or more selected centroids to identify one or more second nearest vectors; and selecting the one or more first nearest vectors based on a comparison of a first distance between the query vector and the one ormore first nearest vectors and a second distance between the query vector and the one or more second nearest vectors.

[0021] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the performing the first ANN search in the first individual hierarchical graph-based indexing database is concurrent with performing the second ANN search in the second individual hierarchical graph-based indexing database.

[0022] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the method further comprises receiving a second query associated with a first centroid of the plurality of centroids; and responding to the second query by providing data objects corresponding to a respective cluster of the plurality of clusters of vectors associated with the first centroid.

[0023] A third aspect of the embodiments of the present disclosure relates to an apparatus comprising a processor, and a memory storing program instructions that, when executed by the processor, cause the apparatus to perform the method of any of the disclosed embodiments.

[0024] A fourth aspect of the embodiments of the present disclosure relates to a non-transitory computer readable medium comprising a computer program product for use by a computer system, the computer program product comprising computer executable instructions stored on the non- transitory computer readable medium that, when executed by one or more processors, cause the computer system to execute the method of any of the disclosed embodiments.

[0025] A fifth aspect of the embodiments of the present disclosure relates to an apparatus comprising means for performing the method of any of the disclosed embodiments.

[0026] For the purpose of clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0027] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0028] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0029] FIG. 1 is a schematic diagram of an example network system that utilizes a map-reduce vector database for query-response according to an embodiment of the present disclosure.

[0030] FIG. 2 is a schematic diagram illustrating an example structure of a hierarchical navigational small world (HNSW) according to an embodiment of the present disclosure.

[0031] FIGS. 3A-3B illustrate an example method of searching a map-reduce vector database according to an embodiment of the present disclosure.

[0032] FIG. 4 is a flowchart of an example method of generating a map-reduce vector database for responding to a query according to an embodiment of the present disclosure.

[0033] FIG. 5 is a flowchart of an example method of searching a map-reduce vector database in response to a query according to an embodiment of the present disclosure.

[0034] FIG. 6 is a block diagram of an example computer apparatus according to an embodiment of the present disclosure.DETAILED DESCRIPTION

[0035] It should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0036] The following terms are defined as follows unless used in a contrary context herein. Specifically, the following definitions are intended to provide additional clarity to the present disclosure. However, terms may be described differently in different contexts. Accordingly, the following definitions should be considered as a supplement and should not be considered to limit any other definitions of descriptions provided for such terms herein.

[0037] While large-language models (LLMs) can be used in many fields to ease and / or advance various operations in businesses, organizations, industries, and / or educational fields, the operations and / or performance of LLMs may rely heavily on knowledgebases (e.g., large dataset) from which the LLMs can query, reference, or retrieve information when generating responses to queries. Thus, the performance of those knowledgebases in terms of memory usage, computational resource usage, knowledgebase building efficiency, and search efficiency can be important. Thoseknowledgebases may be stored and organized in the form of vector databases. Vector databases index data using vectors. These vectors represent the data in a way that captures their meaning and relationships. As an example, in a text-based vector database, similar words or sentences may have vectors that are close to each other in a vector space. Stated differently, the vectors representing the data may correspond to points in a multi-dimensional space. Thus, the goal of answering a query is to search for the closest vector to the query. In other words, a similarity search is performed in the vector database based on the query.

[0038] There are various approaches in indexing vectors in a vector database. Some examples of vector indexing and search techniques may include, but are not limited to, B-tree (BT), flat and inverted-file indexing (FAI-IVF), fast library for approximate nearest neighbor (FL ANN), hierarchical navigational small world (HNSW), knowledge graph (KG), nearest neighbor descent (NND), and small-world graph (SWG) algorithms. Generally, there is a tradeoff among index time, recall, and performance. Index time may refer to the time it takes to construct or build vector indices that enable searches in a vector database. Recall may refer to the accuracy of search results. Performance may refer to the computational complexity in performing a search (e.g., measured in number of queries per second). Currently, the HNSW algorithm may outperform the other vector index and / or search algorithms for search performance and recall. However, one drawback of the HNSW algorithm is the high computational complexity for vector indexing. Furthermore, there is no built-in mechanism for record updates in the HNSW algorithm, making it unsuitable for applications where new data may arrive and real-time record update is desired or needed.

[0039] Disclosed herein are techniques for performing efficient vector indexing and search (e.g., for LLM processing and / or any artificial intelligence (Al) model processing). More specifically, a map-reduce approach is used to reduce the complexity for building and searching a vector database by clustering vectors (e.g., embedding vectors) into multiple clusters (e.g., N clusters, where N is an integer value greater than 1) prior to applying a hierarchical navigational small world (HNSW) algorithm (e.g., an approximate nearest neighbor (ANN) algorithm) to each cluster independently. Stated differently, the map-reduce approach may include clustering vectors to centroids and sharding (e.g., mapping or associating) the centroids to HNSW index shards (e.g., HNSW index databases). A centroid may refer to a geo-centroid (a center point) of a cluster in a multi-dimensional vector space. During a search for a query, the map-reduce approach may locate N 1 most relevant clusters (e.g., closest to the query) and may only search the HNSW index shard(s)corresponding to those N1 most relevant clusters. The map-reduce approach may retrieve K nearest-neighbors from each HNSW index shard and reduce the N1 sets of K nearest-neighbors to K nearest-neighbors.

[0040] According to an embodiment of the present disclosure, a system (e.g., a computer system, a server) for building a map-reduce vector database may receive a plurality of data objects associated with a dataset. The data objects may be text (e.g., documents) or any suitable form of content (e.g., text, audio, and / or video). The system may calculate a plurality of vectors, each representative of one of the plurality of data objects. In an embodiment, each of the plurality of vectors may comprise embeddings (e.g., encoded values) associated with attributes of the respective vector. In an example, the embeddings are generated based on a semantic analysis of the data objects. Next, the system may form a plurality of clusters (e.g., N clusters) from the vectors based on a similarity (or relationships) between the vectors, where each cluster may include a different subset of the vectors. As an example, the data objects may be documents, and the clusters may correspond to different subjects or topics. For instance, one cluster may include documents related to storage, another cluster may include documents related to compute, and yet another cluster may include documents related to network, etc.

[0041] The system may calculate a centroid for each individual cluster based on a respective subset of the vectors. In an embodiment, the centroid for an individual cluster may be an average (or a mean) of the respective subset of the vectors. The system may index a respective subset of the vectors in each individual cluster based on an approximate nearest neighbor (ANN) search to generate an individual hierarchical graph-based indexing database. That is, for N clusters, there may be N number of independent hierarchical graph-based indexing databases, each corresponding to a respective one of the N clusters. In an embodiment, the ANN search is based on an HNSW algorithm. The system may build the map-reduce vector database by storing each centroid in association with a corresponding hierarchical graph-based indexing database, where each hierarchical graph-based indexing database may reference (or link to) a respective subset of the vectors. Further, each vector may reference (or link to) a respective data object. Because each hierarchical graph-based indexing database is independent from each other, the system may generate the hierarchical graph-based indexing databases concurrently using parallel processing, thereby providing efficient vector indexing for large data. For instance, the map-reduce approach may improve vector indexing efficiency by about N folds compared to the HNSW algorithm.

[0042] In an embodiment, the system may receive a new data object after the map-reduce vector database is built. To update the map-reduce vector database with the new data object, the system may calculate a subsequent vector based on attributes of the new data object. The system may insert the subsequent vector into a first cluster of the plurality of clusters based on a distance between the subsequent vector and a respective centroid of the first cluster satisfying a threshold (e.g., below the threshold). As part of the inserting, the system may update a respective individual hierarchical graph-based indexing database corresponding to the first cluster. In some embodiments, the system may determine that the distance between the subsequent vector and the centroid of at least one other cluster (e.g., a second cluster of the plurality of clusters) satisfies the threshold. In such embodiments, the system may search the hierarchical graph-based indexing databases for each of the first and second clusters to identify a nearest-neighboring vector to the subsequent vector and the inserting the subsequent vector into the first cluster may be based on a first distance between the subsequent vector and a first nearest-neighboring vector in the first cluster being shorter than a second distance between the subsequent vector and a second nearest-neighboring vector in the second cluster.

[0043] To search a map-reduce vector database constructed as discussed above for a query, a system may calculate a query vector based on attributes of the query. The system may calculate a distance between the query vector and each centroid in the map-reduce vector database. The system may select one or more centroids from the centroids in the map-reduce vector database based on the distance between the query vector and a respective centroid satisfying a certain threshold (e.g., a configurable threshold). The system may perform an ANN search in each individual hierarchical graph-based indexing databases associated with each of the one or more selected centroids to identify, from a respective one of the clusters of vectors, one or more nearest- neighboring vectors to the query vector. In an embodiment, the system may select more than one centroid (e.g., N1 number of centroids). Thus, the system may select top K nearest-neighboring vectors from each respective hierarchical graph-based indexing databases. That is, N1 sets of K nearest-neighboring vectors may be selected. The system may reduce the N1 sets of K nearest- neighboring vectors by selecting the set of K nearest-neighboring vectors with the shortest distance from the query vector. Next, the system may generate a response to the query based on the identified one or more nearest-neighboring vectors.

[0044] In a further embodiment, the map-reduce vector approach can also allow for queries using a centroid as a search term. For example, the data objects in the clusters are text documents, where one cluster may include documents related to storage with a centroid representing the term “storage”, one cluster may include documents related to compute with a centroid representing the term “compute”, and one cluster may include documents related to networking with a centroid representing the term “networking”. Thus, in an example, a query can be made to request all documents related to storage, and a response to such a query may include all documents in the cluster of documents related to storage.

[0045] Utilizing a map-reduce vector approach to cluster vectors associated with a dataset into multiple clusters prior to applying an HNSW algorithm to each cluster independently can allow for parallel processing during vector indexing and / or during a query search. For instance, during vector indexing, the hierarchical graph-based indexing databases for N plurality of clusters of vectors can be generated at the same time, thereby significantly improving indexing time or database generation time (e.g., by about N folds compared to the HNSW algorithm). During a query search, the search can reduce to searching only the most relevant clusters (or corresponding hierarchical graph-based indexing databases) instead of searching the entire dataset and can further search the most relevant clusters at the same time, thereby further significantly improving search efficiency (e.g., by more than N folds compared to the HNSW algorithm). Additionally, the map- reduce vector approach can support adding new data to a map-reduce vector database. Further, because a centroid is calculated for each cluster, a query can be made using the centroid as a search term to obtain all data objects within a certain cluster. Further, because each hierarchical graphbased indexing database only includes indices for a cluster of vectors rather than the entire dataset, the number of relationships or connections related to a single node may be significantly reduced, and thus the per node memory usage may be significantly reduced.

[0046] FIG. 1 is a schematic diagram of an example network system 100 that utilizes a map- reduce vector database 110 for query-response according to an embodiment of the present disclosure. As shown in FIG. 1, the network system 100 may include a dataset 102, a map-reduce vector database 110, a network 120, a computer system 130, and a client device 106. The network 120 may promote communication between the components of the network system 100. The network 120 may be any communication network including a public data network (PDN), a public switched telephone network (PSTN), a private network, and / or a combination.

[0047] The computer system 130 may include a database generation module 132 and a vector search module 134. The database generation module 132 may include hardware and / or software components configured to build the map-reduce vector database 110 from the dataset 102. In an example, the database generation module 132 may include an application (e.g., a software program) including instructions stored in the memory of the computer system 130 and executable by processor(s) of the computer system 130. The vector search module 134 may include hardware and / or software components configured to search the map-reduce vector database 110 to generate responses to queries (e g., associated with the dataset 102). In an example, the vector search module 134 may include an application (e.g., a software program) including instructions stored in the memory of the computer system 130 and executable by processor(s) of the computer system 130.

[0048] At a high level, a user 107 of the client device 106 (e.g., a laptop, a computer workstation, a tablet, a smartphone, etc.) may ask a question, and the vector search module 134 may generate an answer to the question based on a search in the map-reduce vector database 110. In some examples, the computer system 130 may further include an Al agent application including instructions stored in the memory of the computer system 130 and executable by a processor of the computer system 130. The Al agent application may interact with the user 107 of the client device 106 and the vector search module 134. For instance, the user 107 may ask a question using natural language (e.g., human-like language) and the Al agent application may interact with the vector search module 134 to generate a natural-language response to the question.

[0049] The dataset 102 may include a plurality of data objects 104. In an example, the data objects 104 are text documents. Generally, the data objects 104 may include any suitable form of content (e.g., text, audio, and / or video). To generate the map-reduce vector database 110, the database generation module 132 may convert the data objects 104 into numerical representations in a high-dimensional space (e.g., with 400, 500, 600, 700 or more dimensions). For instance, each data object 104 is converted into a multi-dimensional vector 116 including embeddings (e.g., encoded values) representative of attributes of the respective data object 104. As an example, when the data objects 104 are text-based, the embeddings may be representative of words, phrases, and / or sentences. As another example, when the data objects 104 are images, the embeddings may be representative of features extracted from the images. As yet another example, when the dataobjects 104 are audios, the embeddings may be representative of spectrograms calculated from the audios.

[0050] The database generation module 132 may form a plurality of clusters 112 from the vectors 116. Stated differently, the database generation module 132 may partitions the vectors 116 into a plurality of clusters 112. The clusters 112 are shown as 112a, 112b, ..., 112c. For ease of illustrations, FIG. 1 only illustrates the vectors 116 in the cluster 112c. Further, the vectors 116 are illustrated as data points within the cluster 112c. Generally, each cluster 112 may include a different subset of the vectors 116 converted from the data objects 104. The database generation module 132 may calculate a centroid 114 for each individual cluster 112. As described above, a centroid 114 may refer to a geo-centroid (a center point) of a cluster 112 in a multi-dimensional vector space. For instance, the cluster 112a may have a centroid 114a, the cluster 112b may have a centroid 114b, and so on. Each centroid 114 is a vector in the multi-dimensional vector space. In an embodiment, the database generation module 132 may calculate a centroid 114 for a cluster 112 based on an average (or a mean) of a respective subset of the vectors 116 in the cluster 112. In some examples, the database generation module 132 may form the clusters 112 and / or calculate the respective centroids 114 using an Al model (e.g., a K-mean clustering model).

[0051] The database generation module 132 may map each centroid 114 to a corresponding HNSW index database 118 as shown by the mapping 117. The HNSW index databases 118 are shown as 118a, 118b, . .., 118c. As shown, the centroid 114a of the cluster 112a is mapped to (or associated with) the HNSW index database 118a, the centroid 114b of the cluster 112b is mapped to (or associated with) the HNSW index database 118b, and the centroid 114c of the cluster 112c is mapped to (or associated with) the HNSW index database 118c. The database generation module 132 may generate each individual HNSW index database 118 by indexing a respective subset of the vectors 116 in the corresponding cluster 112. That is, the database generation module 132 may generate the HNSW index database 118a by indexing a respective subset of the vectors 116 in the corresponding cluster 112a, generate the HNSW index database 118b by indexing a respective subset of the vectors 116 in the corresponding cluster 112b, and so on. Each of the HNSW index database 118b is independent of each other. As will be discussed more fully below with reference to FIG. 2, the HNSW indexing scheme may allow for an efficient query search.

[0052] In an embodiment, the map-reduce vector database 110 may be stored in memory as a data structure. In some instances, the map-reduce vector database 110 may be stored on a serverseparate from the computer system 130. In other instances, the map-reduce vector database 1 10 may be stored in the memory of the computer system 130. The map-reduce vector database 110 may generally be arranged in memory in any suitable way. As an example, a map-reduce vector database data structure may store a list of centroids 114 (e.g., vectors in the multi-dimensional vector space) and each centroid 114 may be linked to (e.g., in the form of pointers) a corresponding HNSW index database 118. Stated differently, the map-reduce vector database 110 may store the centroids 114 of the clusters 112 in association with the corresponding HNSW index database 118.

[0053] FIG. 2 is a schematic diagram illustrating an example structure of an HNSW index database 200 according to an embodiment of the present disclosure. In an embodiment, the HNSW index database 200 may correspond to an individual HNSW index database 118 of FIG. 1. As shown in FIG. 2, the HNSW index database 200 may include a plurality of layers 202. For ease of illustration, FIG. 2 only illustrates three layers 202a, 202b, and 202c. However, an HNSW index database 200 may generally include any suitable number of layers 202 (e.g., 2, 3, 4, 5, 6, 7, 8, 9, 10 or more). Each layer 202 may include a plurality of nodes 204 interconnected by respective connections 206 (or links). For ease of illustration, FIG. 2 only illustrates one node with the label 204 and one connection with the label 206. Each node 204 may correspond to a vector (e.g., a vector 116) within a cluster (e.g., a cluster 112). A connection 206 interconnecting two nodes 204 may represent a relationship between the two nodes 204. For instance, the relationship may be a similarity measure between the two nodes 204, where the longer the connection 206, the lower the similarity between the two nodes 204. The connections 206 are shown as solid lines when the respective connections connect nodes 204 in the same layer 202. The connections 206 are shown as dashed lines when the respective connections connect nodes 204 across different layers 202. Further, the connections 206 with arrows are intended to illustrate an example search in the HNSW index database 200 as will be discussed more fully below.

[0054] The HNSW index database 200 may be arranged such that the topmost layer 202 (e.g., 202a) may have the fewest number nodes 204 and the longest connections 206, while the bottommost layer 202 (e.g., 202c) may have all the nodes 204 (corresponding to the vectors 116 representing all the data objects 104 in the dataset 102) and the shortest connections 206. For instance, the layers 202 may include an increasing number of nodes 204 and decreasing connection distances from a top layer 202a to a bottom layer 202c. Such an arrangement may allow for an efficient search for similar vectors in a large dataset. The process of inserting a new vector into theHNSW index database 200 may be substantially similar to the process of searching for a query in the HNSW index database 200. The following provides an example search in the HNSW index database 200, followed by an example construction of the HNSW index database 200.

[0055] A greedy search approach may be used to search the HNSW index database 118 for a node 204 closest to a query vector 210. The search may begin at the topmost layer 202a. A starting node 204 may be arbitrarily or randomly selected. The greedy search may move down layer 202 by layer 202, each time choosing the nearest-neighboring node 204 in the connected subset of nodes 204 in that layer 202. This process may continue until the search reaches the bottommost layer 202c, where the search may return a set of top K nearest-neighboring nodes 204 for the query vector 210. In FIG. 2, the entry nodes 204 at each layer 202 are shown by solid-filled circles, the visited nodes 204 are shown by circles with diagonal stripes, unvisited nodes 204 are shown by empty-filled circles, and the nearest-neighboring node 204 at each layer 202 is shown by circles with a diamond grid. As shown in FIG. 2, at each layer 202, the search may explore (or visit) a list of top K neighboring nodes 204 and stop when no closer neighboring node 204 is found. The discovered closest-neighboring node 204 on the current layer 202 may be treated as the starting point for searching in the next lower layer 202. For instance, after finding the closest-neighboring node 204 on the current layer 202, the search may traverse to the next lower layer 202. That is, the next lower layer 202 may become the new current layer 202. On the new current layer 202, the search may select the nearest-neighboring node 204 found from the upper layer 202 as the starting point (shown by the dashed lines connecting nodes 204 from one layer 202 to a next lower layer 202). The search may move towards the query vector 210 each time by expanding the number of neighboring nodes 204. The list of top K neighboring nodes 204 may be updated as long as any closer top K neighboring node 204 is found. The recall (or accuracy) of the search procedure is controlled by the parameter K. In an example, the parameter K may be configurable (e.g., a value of 1, 2, 3, 4, 5, 6, 7, 8 or more). Generally, the larger the dataset size, the greater the value for the parameter K.

[0056] Construction of the HNSW index database 200 may follow a substantially similar procedure as the nearest-neighbor search discussed above. Each new vector (e.g., a new vector 116 corresponding to a new data object 104) may be treated as a query against the HNSW index database 200 under construction. The HNSW index database 200 may be built incrementally by repetitively inserting vectors into the hierarchy. The probability that a query is inserted into a layer202 is regularized by an exponentially decaying probability distribution. The lower the layer 202 is, the higher the chance that the query is inserted. On the layer 202 that a vector sample is to be inserted, the discovered M neighboring nodes 204 are kept in a neighbor list for the sample. Accordingly, the vector is possibly inserted into the neighbor list of these M neighboring nodes 204. The parameter M may control the scale of the neighbor list. In an example, the parameter M may also be configurable (e.g., a value of 3, 4, 5, 6, 7, 8, 8, 10 or more). Generally, the larger the dataset size, the greater the value for the parameter M.

[0057] FIGS. 3A-3B are discussed in relation to FIGS. 1 and 2. FIGS. 3A-3B illustrate an example method 300 of generating a response to a query using a map-reduce vector database 110 according to an embodiment of the present disclosure. The method 300 may be performed by the vector search module 134. As shown in FIG. 3 A, the vector search module 134 may receive a query 302 (e g., received from a client device 106). At operation 310, the vector search module 134 may calculate a query vector 304 based on attributes of the query 302. For example, the query vector 304 may include embeddings representing attributes of the query 302. As an example, when the query 302 is text -based, the embeddings may be representative of words, phrases, and / or sentences. As another example, when the query 302 includes an image, the embeddings may be representative of features extracted from the images. As yet another example, when the query 302 includes an audio, the embeddings may be representative of spectrograms calculated from the audios.

[0058] Next, the vector search module 134 may select one or more centroids 1 14 from the list of centroids 114a, 114b, . . . , 114c. In the illustrated example of FIG. 3 A, the vector search module 134 may select the centroids 114a and 114b. The selection may be based on a distance between the query vector 304 and each of the one or more centroids 114. For instance, the vector search module 134 may compute a distance between the query vector 304 and each of the one or more centroids 114 and apply a threshold to each computed distance.

[0059] FIG. 3B provides an example query-centroid distance calculation. As shown in FIG. 3B, the query vector 304 may be located between the clusters 112a and 112b. For ease of illustration, the clusters 112a and 112b in FIG. 3B are shown as 2-dimensional. However, the clusters 112a and 112b may have any suitable number of dimensions (e.g., 400, 500, 600, 700 or more) as discussed above. The vector search module 134 may determine that the distance 306a between the query vector 304 and the centroid 114a (of the cluster 112a) is less than the threshold.The vector search module 134 may determine that the distance 306b between the query vector 304 and the centroid 114b (of the cluster 112b) is also less than the threshold. Thus, the vector search module 134 may search, in each of the clusters 112a and 112b, for K vectors 116 nearest to the query vector 304.

[0060] Returning to FIG. 3A, to search for K nearest vectors 116 in the cluster 112a, the vector search module 134 may search for K nearest-neighboring nodes (e.g., nodes 204) in the HNSW index database 118a. Similarly, to search for K nearest vectors 116 in the cluster 112b, the vector search module 134 may search for K nearest-neighboring nodes (e.g., nodes 204) in the HNSW index database 118b. The search for K the nearest -neighboring nodes 204 in each of the individual HNSW index databases 118a and 118b may be as discussed above with reference to FIG. 2. For instance, the vector search module 134 may find a first set of K nearest-neighboring nodes 204 (or corresponding vectors 116) from the HNSW index databases 118a and a second set of K nearest- neighboring nodes 204 (or corresponding vectors 116) from the HNSW index databases 118b.

[0061] At operation 320, the vector search module 134 may reduce the two sets of K nearest- neighboring vectors 116 to K nearest-neighboring vectors 116. For instance, the vector search module 134 may select, from the first set or the second set, the set of K nearest-neighboring vectors 116 with the shortest distance from query vector 304. At operation 330, the vector search module 134 may generate a response to the query 302 based on the reduced K nearest-neighboring vectors 116. For instance, the response may be generated based on the data objects 104 corresponding to the reduced K nearest-neighboring vectors 116.

[0062] Generally, the vector search module 134 may select N1 clusters 112 based on the distance between each respective centroid 114 and the query vector 304 satisfying the threshold. After selecting N1 clusters 112, the vector search module 134 may search for K nearest- neighboring vectors 116 in each of the N1 clusters 112 by searching the associated HNSW index database 118 (e.g., using mechanisms discussed above with reference to FIG. 2). Subsequently, the vector search module 134 may select the K nearest-neighboring vectors 116 with the shortest distance from the query vector 304 among the N1 sets of K nearest-neighboring vectors 116.

[0063] In an embodiment, the map-reduce vector database 110 may be updated as new data record or data object 104 is available. The updating of the map-reduce vector database 110 may use substantially similar mechanisms as searching the map-reduce vector database 110 discussed above with reference to FIGS. 3A-3B. For instance, the database generation module 132 mayreceive a new data object 104. The database generation module 132 may calculate a new vector (e.g., a vector 116) including embeddings representing attributes of the new data object 104. The database generation module 132 may compute a distance between the new vector and each of the centroids 114 and apply a threshold to each computed distance. If there is only one centroid 114 with a distance to the new vector satisfying the threshold, the database generation module 132 may insert the new vector into the cluster 112 corresponding to that centroid 114. The inserting may include updating the HNSW index database 118 corresponding to (or associated with) that cluster 112 (e.g., using the HNSW construction mechanisms as discussed above with reference to FIG. 2). If, however, there are two or more centroids 114 with respective distances to the new vector satisfying the threshold, the database generation module 132 may search each of the HNSW index databases 118 associated with the two or more centroids 114 for a vector 116 closest to the new vector. The database generation module 132 may select the HNSW index database 118 (or cluster 112) with the nearest vector having the shortest distance to the new vector among the nearest vectors found in the HNSW index databases 118 associated with the two or more centroids 114. The database generation module 132 may update the selected HNSW index database 118 with the new vector.

[0064] FIG. 4 is a flowchart of an example method 400 of generating a map-reduce vector database (e.g., the map-reduce vector database 110) for responding to query according to an embodiment of the present disclosure. In embodiments, the method 400 may be implemented by the database generation module 132. In embodiments, the method 400 may be implemented using a computer system with components as shown in FIG. 6. The method 400 may use similar mechanisms as discussed above with reference to FIGS. 1-2 and 3A-3B. As illustrated, FIG. 4 includes a number of enumerated operations, but embodiments of the operations in FIG. 4 may include additional operations before, after, and in between the enumerated operations. In some embodiments, one or more of the enumerated operations may be omitted or performed in a different order.

[0065] At operation 402, a plurality of data objects (e.g., the data objects 104) associated with a dataset (e.g., the dataset 102) is received. The data objects may include any suitable form of content (e.g., text, audio, and / or video). At operation 404, a plurality of vectors (e.g., the vectors 116), each representative of a respective one of the plurality of data objects are calculated. In an embodiment, a first vector of the plurality of vectors includes one or more embeddings associatedwith one or more attributes of a respective one of the plurality of data objects. In an embodiment, the one or more embeddings in the first vector are based on a semantic analysis of the respective one of the plurality of data objects (e.g., when the data objects include text). In another embodiment, the one or more embeddings in the first vector are based on a feature analysis of the respective one of the plurality of data objects (e.g., when the data objects include images). In yet another embodiment, the one or more embeddings in the first vector are based on a spectrogram or audio analysis of the respective one of the plurality of data objects (e.g., when the data objects include audios).

[0066] At operation 406, a plurality of clusters (e.g., the clusters 112) are formed based on a similarity between the plurality of vectors (e.g., the vectors 116), where each cluster of the plurality of clusters includes a different subset of the plurality of vectors. In some instances, the similarity is measured based on Euclidean distances or cosine distances among the plurality of vectors.

[0067] At operation 408, a centroid (e g., the centroids 114) for an individual cluster of the plurality of clusters is calculated based on a respective subset of the plurality of vectors in the cluster. In an embodiment, the calculating the centroid for the individual cluster is further based on an average (or mean) of the respective subset of the plurality of vectors within the individual cluster.

[0068] At operation 410, the respective subset of the plurality of vectors in the individual cluster is indexed based on an ANN search to generate an individual hierarchical graph-based indexing database (e.g., the HNSW index databases 118 and 200). In an embodiment, the individual hierarchical graph-based indexing database for the individual cluster includes a plurality of layers (e.g., the layers 202) including a plurality of nodes (e.g., the nodes 204) corresponding to the respective subset of the plurality of vectors, and a first layer of the plurality of layers includes a fewer number of nodes connected by one or more longer distance connections (e.g., the connections 206) than a second layer of the plurality of layers (e.g., as discussed above with reference to FIG. 2). For instance, the first layer may be a higher layer than the second layer in a multi-level hierarchy. In an embodiment, the ANN search used for indexing the respective subset of the plurality of vectors in each cluster to generate the individual hierarchical graph-based indexing database is based on an HNSW algorithm (e.g., as discussed above with reference to FIG. 2). In an embodiment, the indexing the respective subset of the plurality of vectors in the individual cluster based on the ANN search includes generating a first hierarchical graph-based indexing database for a first cluster of the plurality of clusters concurrent with generating a second hierarchical graph-based indexing database for a second cluster of the plurality of clusters. As discussed above, because each of the hierarchicalgraph-based indexing databases is independent of each other, the generation of the hierarchical graph-based indexing databases may be performed in parallel (e.g., using parallel processing) to provide efficient vector or database indexing.

[0069] At operation 412, the centroid of the individual cluster is stored in a database (e.g., the map-reduce vector database 110) in association with the corresponding individual hierarchical graph-based indexing database. At operation 414, a query (e.g., the query 302) associated with the dataset is received. At operation 416, the query is responded to based on the database (e.g., based on a search for the nearest vector in the database).

[0070] In an embodiment, the method 400 further includes receiving a subsequent data object (e.g., a new data object 104). The method 400 further includes calculating a subsequent vector (e.g., a new vector 116) representative of one or more attributes of the subsequent data object. The method 400 further includes inserting the subsequent vector into a first cluster of the plurality of clusters based on a distance (e.g., the distances 306a and 306b) between the subsequent vector and a respective centroid of the first cluster satisfying a threshold, (e.g., twice the radius of a cluster). Further, the inserting includes updating a respective individual hierarchical graph-based indexing database corresponding to the first cluster (e.g., using mechanisms as discussed above with reference to FIGS. 2 and 3A-3B). In an embodiment, the inserting the subsequent vector into the first cluster is further based on a comparison of a first distance between the subsequent vector and a first nearest- neighbor in the first cluster and a second distance between the subsequent vector and a second nearest-neighbor in a second cluster of the plurality of clusters (e.g., the first distance is shorter than the second distance).

[0071] FIG. 5 is a flowchart of an example method 500 of searching a map-reduce vector database (e.g., the map-reduce vector database 110) in response to a query (e.g., the query 302) according to an embodiment of the present disclosure. In embodiments, the method 500 may be implemented by the vector search module 134. In embodiments, the method 500 may be implemented using a computer system with components as shown in FIG. 6. The method 500 may use similar mechanisms as discussed above with reference to FIGS. 1-2 and 3A-3B. As illustrated, FIG. 5 includes a number of enumerated operations, but embodiments of the operations in FIG. 5 may include additional operations before, after, and in between the enumerated operations. In some embodiments, one or more of the enumerated operations may be omitted or performed in a different order.

[0072] At operation 502, a query (e.g., the query 302) associated with a dataset (e.g., the dataset 102) is received. At operation 504, a query vector (e.g., the query vector 304) is calculated based on one or more attributes of the query.

[0073] At operation 506, one or more centroids (e.g., the centroids 114) are selected from a plurality of centroids of respective ones of a plurality of clusters (e.g., the clusters 112) of vectors (e.g., the vectors 116). The selecting is based on a distance (e.g., the distances 306a and 306b) between the query vector and each of the one or more centroids. Each vector of the plurality of clusters of vectors is representative of a respective one of the plurality of data objects in the dataset. In an embodiment, a first vector of the plurality of vectors includes embeddings associated with one or more attributes of a respective one of the plurality of data objects (e.g., including text, image, and / or audio). In an embodiment, the one or more embeddings in the first vector are based on a semantic analysis of the respective one of the plurality of data objects (e.g., when the data objects include text). In another embodiment, the one or more embeddings in the first vector are based on a feature analysis of the respective one of the plurality of data objects (e.g., when the data objects include images). In yet another embodiment, the one or more embeddings in the first vector are based on a spectrogram or audio analysis of the respective one of the plurality of data objects (e.g., when the data objects include audios). In an embodiment, a first centroid of the plurality of centroids is an average (or mean) of a respective cluster of the plurality of clusters of vectors. In an embodiment, the selecting is further based on the distance between the query vector and each of the one or more centroids satisfying a threshold.

[0074] At operation 508, an ANN search is performed, based on the query vector, in each individual hierarchical graph-based indexing database (e.g., theHNSW index databases 118 and / or 200) associated with each of the one or more selected centroids to identify, from a respective one of the plurality of clusters of vectors, one or more nearest vectors (e.g., K nearest-neighboring vectors) to the query vector. In an embodiment, each individual hierarchical graph-based indexing database associated with each of the one or more selected centroid includes a plurality of layers (e g., the layers 202) including a plurality of nodes (e.g., the nodes 204) corresponding to a respective cluster of vectors, and a first layer of the plurality of layers includes a fewer number of nodes connected by one or more longer distance connections (e.g., the connections 206) than a second layer of the plurality of layers. For instance, the first layer may be a higher layer than the second layer in a multi-level hierarchy. In an embodiment, each individual hierarchical graph-basedindexing database associated with each of the one or more selected centroids is based on an HNSW algorithm.

[0075] In an embodiment, the performing the ANN search in each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids includes performing, based on the query vector, a first ANN search in a first individual hierarchical graphbased indexing database (e.g., the HNSW index database 118a) associated with a first centroid (e.g., the centroid 114a) of the one or more selected centroids to identify one or more first nearest vectors and performing, based on the query vector, a second ANN search in a second individual hierarchical graph-based indexing database (e.g., the HNSW index database 118b) associated with a second centroid (e.g., the centroid 114b) of the one or more selected centroids to identify one or more second nearest vectors. The performing the ANN search further includes selecting the one or more first nearest vectors based on a comparison of a first distance between the query vector and the one or more first nearest vectors and a second distance between the query vector and the one or more second nearest vectors. For instance, the selecting the one or more first nearest vectors is based on the first distance being shorter than the second distance. In an embodiment, the performing the first ANN search in the first individual hierarchical graph-based indexing database is concurrent with performing the second ANN search in the second individual hierarchical graphbased indexing database. As discussed above, because each of the hierarchical graph-based indexing databases is independent of each other, the searching of the hierarchical graph-based indexing databases may be performed in parallel (e.g., using parallel processing) to provide efficient vector or database search.

[0076] At operation 510, a response to the query is generated based on the identified one or more nearest vectors. For instance, the response may be generated based on the data objects corresponding to the identified one or more nearest vectors.

[0077] In an embodiment, the method 500 further includes receiving a second query associated with a first centroid of the plurality of centroids (e.g., using the first centroid as a query term). The method 500 further includes responding to the second query by providing data objects corresponding to a respective cluster of the plurality of clusters of vectors associated with the first centroid. As an example, the data objects in the clusters are text documents, where one cluster may include documents related to storage with a centroid representing the term “storage”, one cluster may include documents related to compute with a centroid representing the term “compute”, andone cluster may include documents related to networking with a centroid representing the term “networking”. Thus, in an example, a query can be made to request all documents related to storage, and a response to such a query may include all documents in the cluster of documents related to storage.

[0078] FIG. 6 is a schematic diagram of a computer apparatus 600 (e.g., part of the computer system 130). The computer apparatus 600 is suitable for implementing the disclosed embodiments as described herein. The computer apparatus 600 comprises ingress ports / ingress means 610 (a.k.a., upstream ports) and receiver units (Rx)Zreceiving means 620 for receiving data; a processor, logic unit, or central processing unit (CPU)Zprocessing means 630 to process the data; transmitter units (Tx)Ztransmitting means 640 and egress portsZegress means 650 (a.k.a., downstream ports) for transmitting the data; and a memoryZmemory means 660 for storing the data. The computer apparatus 600 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress portsZingress means 610, the receiver unitsZreceiving means 620, the transmitter unitsZtransmitting means 640, and the egress portsZegress means 650 for egress or ingress of optical or electrical signals.

[0079] The processorZprocessing means 630 is implemented by hardware and software. The processorZprocessing means 630 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processorZprocessing means 630 is in communication with the ingress portsZingress means 610, receiver unitsZreceiving means 620, transmitter unitsZtransmitting means 640, egress portsZegress means 650, and memoryZmemory means 660. The processorZprocessing means 630 comprises a map-reduce vector module 670. The map-reduce vector module 670 is able to implement the methods disclosed herein. For example, in one embodiment, the map-reduce vector module 670 may implement a generation of a map- reduce vector database (e.g., the map-reduce vector database 110) andZor an update of the map- reduce vector database. In another embodiment, the map-reduce vector module 670 may implement a query search in a map-reduce vector database. The inclusion of the map-reduce vector module 670 therefore provides a substantial improvement to the functionality of the computer apparatus 600 and effects a transformation of the computer apparatus 600 to a different state. Alternatively, the map-reduce vector module 670 is implemented as instructions stored in the memoryZmemory means 660 and executed by the processorZprocessing means 630.

[0080] The computer apparatus 600 may also include input and / or output (I / O) devices or I / O means 680 for communicating data to and from a user. The I / O devices or I / O means 680 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices or I / O means 680 may also include input devices, such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0081] The memory / memory means 660 comprises one or more disks, tape drives, and solid- state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory / memory means 660 may be volatile and / or non-volatile and may be readonly memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0082] It should also be understood that the steps of the exemplary methods set forth herein are not necessarily required to be performed in the order described, and the order of the steps of such methods should be understood to be merely exemplary. Likewise, additional steps may be included in such methods, and certain steps may be omitted or combined, in methods consistent with various embodiments of the present disclosure.

[0083] While several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0084] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the spirit and scope disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising: forming, based on a similarity between a plurality of vectors, a plurality of clusters, wherein each of the plurality of vectors is representative of a respective one of a plurality of data objects in a dataset, and wherein each cluster of the plurality of clusters comprises a different subset of the plurality of vectors; calculating a centroid for an individual cluster of the plurality of clusters based on a respective subset of the plurality of vectors in the cluster; indexing the respective subset of the plurality of vectors in the individual cluster based on an approximate nearest neighbor (ANN) search to generate an individual hierarchical graph-based indexing database; storing, in a database, the centroid of the individual cluster in association with the corresponding individual hierarchical graph-based indexing database; receiving a query associated with the dataset; and responding to the query based on the database.

2. The method of claim 1, wherein a first vector of the plurality of vectors comprises one or more embeddings associated with one or more attributes of a respective one of the plurality of data objects.

3. The method of any of claims 1 -2, wherein the one or more embeddings in the first vector are based on a semantic analysis of the respective one of the plurality of data objects.

4. The method of any of claims 1-3, wherein the calculating the centroid for the individual cluster is further based on an average of the respective subset of the plurality of vectors within the individual cluster.

5. The method of any of claims 1-4, wherein the individual hierarchical graph-based indexing database for the individual cluster comprises a plurality of layers comprising a plurality of nodes corresponding to the respective subset of the plurality of vectors, and wherein a first layer of theplurality of layers comprises a fewer number of nodes connected by one or more longer distance connections than a second layer of the plurality of layers.

6. The method of any of claims 1-5, wherein the ANN search used for indexing the respective subset of the plurality of vectors in the individual cluster to generate the individual hierarchical graph-based indexing database is based on a hierarchical navigational small world (HNSW) algorithm.

7. The method of any of claims 1-6, wherein the indexing the respective subset of the plurality of vectors in the individual cluster based on the ANN search comprises: generating a first hierarchical graph-based indexing database for a first cluster of the plurality of clusters concurrent with generating a second hierarchical graph-based indexing database for a second cluster of the plurality of clusters.

8. The method of any of claims 1-7, further comprising: receiving a subsequent data object; calculating a subsequent vector representative of the subsequent data object; and inserting the subsequent vector into a first cluster of the plurality of clusters based on a distance between the subsequent vector and a respective centroid of the first cluster satisfying a threshold, wherein the inserting comprises updating a respective individual hierarchical graph-based indexing database corresponding to the first cluster.

9. The method of claim 8, wherein the inserting the subsequent vector into the first cluster is further based on a comparison of a first distance between the subsequent vector and a first nearest- neighbor in the first cluster and a second distance between the subsequent vector and a second nearest-neighbor in a second cluster of the plurality of clusters.

10. A computer-implemented method comprising: receiving a query associated with a dataset; calculating a query vector based on one or more attributes of the query;selecting one or more centroids from a plurality of centroids of respective ones of a plurality of clusters of vectors, wherein the selecting is based on a distance between the query vector and each of the one or more centroids, and wherein each vector of the plurality of clusters of vectors is representative of a respective one of a plurality of data objects in the dataset; performing, based on the query vector, an approximate nearest neighbor (ANN) search in each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids to identify, from a respective one of the plurality of clusters of vectors, one or more nearest vectors to the query vector; and generating, based on the identified one or more nearest vectors, a response to the query.

11. The method of claim 10, wherein a first vector of the plurality of clusters of vectors comprises embeddings associated with one or more attributes of a respective one of the plurality of data objects.

12. The method of any of claims 10-11, wherein the one or more embeddings in the first vector are based on a semantic analysis of the respective one of the plurality of data objects.

13. The method of any of claims 10-12, wherein a first centroid of the plurality of centroids is an average of a respective cluster of the plurality of clusters of vectors.

14. The method of any of claims 10-13, wherein the selecting is further based on the distance between the query vector and each of the one or more centroids satisfying a threshold.

15. The method of any of claims 10-14, wherein each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids comprises a plurality of layers comprising a plurality of nodes corresponding to a respective cluster of vectors, and wherein a first layer of the plurality of layers comprises a fewer number of nodes connected by one or more longer distance connections than a second layer of the plurality of layers.

16. The method of any of claims 10-15, wherein each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids is based on a hierarchical navigational small world (HNSW) algorithm.

17. The method of any of claims 10-16, wherein the performing the ANN search in each individual hierarchical graph-based indexing database associated with each of the one or more selected centroids comprises: performing, based on the query vector, a first ANN search in a first individual hierarchical graph-based indexing database associated with a first centroid of the one or more selected centroids to identify one or more first nearest vectors; performing, based on the query vector, a second ANN search in a second individual hierarchical graph-based indexing database associated with a second centroid of the one or more selected centroids to identify one or more second nearest vectors; and selecting the one or more first nearest vectors based on a comparison of a first distance between the query vector and the one or more first nearest vectors and a second distance between the query vector and the one or more second nearest vectors.

18. The method of claim 17, wherein the performing the first ANN search in the first individual hierarchical graph-based indexing database is concurrent with performing the second ANN search in the second individual hierarchical graph-based indexing database.

19. The method of any of claims 10-18, further comprising: receiving a second query associated with a first centroid of the plurality of centroids; and responding to the second query by providing data objects corresponding to a respective cluster of the plurality of clusters of vectors associated with the first centroid.

20. An apparatus comprising: a processor, and a memory storing program instructions that, when executed by the processor, cause the apparatus to perform the method according to any of claims 1-9.

21. An apparatus comprising: a processor, anda memory storing program instructions that, when executed by the processor, cause the apparatus to perform the method according to any of claims 10-19.

22. A non-transitory computer readable medium comprising a computer program product for use by a computer system, the computer program product comprising computer executable instructions stored on the non-transitory computer readable medium that, when executed by one or more processors, cause the computer system to execute the method of any of claims 1-9.

23. A non-transitory computer readable medium comprising a computer program product for use by a computer system, the computer program product comprising computer executable instructions stored on the non-transitory computer readable medium that, when executed by one or more processors, cause the computer system to execute the method of any of claims 10-19.

24. An apparatus comprising means for performing the method of any of claims 1-9.

25. An apparatus comprising means for performing the method of any of claims 10-19.

Images