Machine learning powered search for an object store database
By employing a distributed object store system to co-locate similar vector embeddings and using similarity index computations followed by k-Nearest Neighbors searches, the challenge of efficient similarity searches in object store databases is addressed, achieving fast and precise results for AI-generated data.
Patent Information
- Application Number
- PCT/IB2023/057610
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-07-27
- Publication Date
- 2025-01-30
AI Technical Summary
Existing object store databases lack efficient mechanisms for performing similarity searches across large datasets, particularly for AI-generated vector embeddings, which hinders effective information retrieval and pattern recognition.
A distributed object store system is used to store vector embeddings in a way that similar objects are co-located in the same shard, with a globally unique key combining object and vector identifiers. Similarity searches are performed using a similarity index computation across shards, followed by a k-Nearest Neighbors search in the most similar shard.
This approach enables fast and precise similarity searches across a broad vector space, supporting a wide range of AI/ML models and accommodating high-dimensional data with over 20,000 dimensions.
Description
[0001] Machine Learning Powered Search for an Object Store Database
[0002] Vector Embedding refers to the process of transforming objects such as text, images, video, or audio into numerical representations that reside in a high-dimensional vector space. This technique is achieved through the use of Machine Learning (ML) algorithms that enable the understanding of the meaning and context of data (semantic relationships), learning of complex relationships and patterns within the data (syntactic relationships). Further we can use the resulting vector representations for a wide range of applications such as information retrieval, image classification, natural language processing, and many others. Vector embeddings have become increasingly popular due to their ability to capture semantic meaning and similarities between objects in a way that is easily computable and scalable. Here in this invention, we use a distributed object store system to store the above mentioned vector embeddings into different shards such that vector embeddings of similar objects are co-located in the same physical shard. Further globally unique key is a combination object identifier and vector embedding identifier, and in one of the illustrative embodiments the object identifier is the “prefix” followed by vector embedding identifier forming together the primary key for the distributed object store. Further when a similarity search has to be performed for Artificial Intelligence (Al) generated vector embeddings then first a “similarity index” or “degree of similarity” with a shard of the distributed object store is first quickly computed in a broad vector space consisting of all vector representations of all shards. Note for the above similarity measurement or computation we support Euclidean, cosine similarity, dot product, etc. and can accommodate more than 20,000 dimensions, making it well-suited to support a wide range of foundational and other AI / ML models. Finally in the shard with the highest similarity index, we perform search of related vector embeddings using k-Nearest Neighbours (kNN) algorithm to deliver reliable and precise results.
Claims
Following are the claims for this invention :-1. Vector Embedding refers to the process of transforming objects such as text, images, video, or audio into numerical representations that reside in a highdimensional vector space. This technique is achieved through the use of Machine Learning (ML) algorithms that enable the understanding of the meaning and context of data (semantic relationships), learning of complex relationships and patterns within the data (syntactic relationships). Further we can use the resulting vector representations for a wide range of applications such as information retrieval, image classification, natural language processing, and many others. Vector embeddings have become increasingly popular due to their ability to capture semantic meaning and similarities between objects in a way that is easily computable and scalable. Here in this invention, we use a distributed object store system to store the above mentioned vector embeddings into different shards such that vector embeddings of similar objects are co-located in the same physical shard.
2. Further globally unique key is a combination object identifier and vector embedding identifier, and in one of the illustrative embodiments the object identifier is the “prefix” followed by vector embedding identifier forming together the primary key for the distributed object store.
3. Further when a similarity search has to be performed for Artificial Intelligence (Al) generated vector embeddings then first a “similarity index” or “degree of similarity” with a shard of the distributed object store is first quickly computed in a broad vector space consisting of all vector representations of all shards.
4. Note for the above similarity measurement or computation we support Euclidean, cosine similarity, dot product, etc. and can accommodate more than 20,000 dimensions, making it well-suited to support a wide range of foundational and other AI / ML models.
5. Finally in the shard with the highest similarity index, we perform search of related vector embeddings using k-Nearest Neighbours (kNN) algorithm to deliver reliable and precise results.