Method and System for Searching Nearest Neighbors in a Cardinality-Based Vector Database

The method optimizes ANN search in vector databases by analyzing cardinality data to adjust filtering and search algorithms, improving performance and accuracy in high-dimensional data spaces.

KR103045974B1Active Publication Date: 2026-11-10DINOTICIA CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
KR1020240057959
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-04-30
Publication Date
2026-11-10
Estimated Expiration
2044-04-30

AI Technical Summary

Technical Problem

Existing Approximate Nearest Neighbor (ANN) search methods in vector databases suffer from performance degradation due to high-cardinality filtering, leading to lower recall accuracy and inefficient search performance.

Method used

A method that statistically analyzes cardinality data in a filter condition column, adjusting the search algorithm based on predetermined threshold values to perform KNN search, expand search space when necessary, and utilize HNSW algorithm for improved accuracy and efficiency.

Benefits of technology

Enhances search performance and accuracy by optimizing data filtering and ANN search integration, particularly in high-dimensional data spaces.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112024047635072-PAT00001_ABST
    Figure 112024047635072-PAT00001_ABST
Patent Text Reader

Abstract

A method and system for searching nearest neighbors within a cardinality-based vector database are described. This method includes the steps of statistically analyzing and recording cardinality data in a filter condition column, and determining a data filtering and search method by comparing the recorded statistically analyzed cardinality data with a predetermined set value. If the cardinality data is higher than the predetermined set value, it is determined to perform general data filtering and a k-nearest neighbor (KNN) search on the result. If the cardinality data is lower than the predetermined set value, it is determined not to search neighbor nodes that do not satisfy the filtering condition during the search, and to perform the search by modifying the search algorithm to expand the search space by a multiple of the inverse value of the cardinality or filter combination probability. If the cardinality data is lower than a predetermined threshold value, the nearest neighbor search can be performed using the HNSW algorithm and the search space can be expanded during the greedy search process for the candidate set removed by filtering by increasing the value of ef-search by that multiple, where the predetermined threshold value may be determined as a case where the number of cardinality eigenvalues ​​is 25 or more, or a case where the joint probability of the filter condition is 4% or less.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a vector database, and more specifically, to a method and system for performing nearest neighbor search in a vector database. Background Technology

[0002] In machine learning or artificial intelligence, the process of converting human-perceived data into AI-perceived data, primarily text, voice, and video, into vector data is called data embedding.

[0003] Vector databases are databases that store vector data and search based on the similarity between stored vectors, and are widely used in the field of artificial intelligence.

[0004] In such vector databases, Nearest Neighbor search refers to the process of finding the data point closest to a given point in a multidimensional space. This method plays an important role in various fields, including machine learning, databases, and information retrieval. In particular, Approximate Nearest Neighbor (ANN) search is a technique that sacrifices some accuracy to reduce computational costs and is widely used due to its efficiency in large-scale datasets. In other words, since ANNs can relax precision to some extent, they enable faster and more scalable search, which is crucial for processing large datasets or high-dimensional data spaces.

[0005] For searching high-dimensional data, measuring the angle between two vectors rather than direct distance is often effective because it handles sparsity better. ANNs utilize techniques such as Locality-Sensitive Hashing (LSH) or Hierarchical Navigable Small Worlds (HNSW) to explore high-dimensional search spaces more efficiently, allowing them to focus on the search and avoid unnecessary computations, thereby accelerating the process and reducing computational requirements.

[0006] Such ANN search considers factors such as the application's acceptable range for the sacrificed parts, the size and dimensions of the dataset, and the quality of data embeddings. For example, ANN search suffers from performance degradation when integrated with data filtering; search performance can vary significantly depending on the filtering conditions of the data, and this depends heavily on the number of eigenvalues ​​(cardinality, hereinafter referred to as cardinality) of the filtering conditions. When high-cardinality filtering is applied during the ANN search process, many neighboring nodes are excluded from the search, resulting in lower recall accuracy; therefore, a method to improve this is required. Prior art literature

[0007] U.S. Patent Publication US20210406321A1 (Filed June 24, 2020) U.S. Patent Application US17 / 582,267 (Filed January 24, 2022) The problem to be solved

[0008] The technical problem of the present invention is to optimize the performance of ANN search by considering the cardinality of filtering conditions in vector database search. means of solving the problem

[0009] According to one aspect of the present invention, a nearest neighbor search method in a cardinality-based vector database comprises the steps of statistically analyzing and recording cardinality data in a filter condition column, and determining a data filtering and search method by comparing the recorded statistically analyzed cardinality data with a predetermined set value. If the cardinality data is higher than the predetermined set value, the determination is made to perform general data filtering and a k-nearest neighbor (KNN) search on the result. If the cardinality data is lower than the predetermined set value, the determination is made not to search neighbor nodes that do not satisfy the filtering condition during the search, and to perform a search by modifying the search algorithm so as to expand the search space by a multiple of the inverse value of the cardinality or filter combination probability. If the cardinality data is lower than a predetermined threshold value, the nearest neighbor search can be performed using the HNSW algorithm and the search space can be expanded during the greedy search process for the candidate set removed by filtering by increasing the value of ef-search by that multiple, where the predetermined threshold value may be determined as a case where the number of cardinality eigenvalues ​​is 25 or more, or a case where the joint probability of the filter condition is 4% or less. Effects of the invention

[0010] According to the present invention, higher performance and accuracy can be achieved in nearest neighbor search after data filtering.

[0011] According to the present invention, higher performance and accuracy can be achieved by optimizing the integration of data filtering and ANN search. Brief explanation of the drawing

[0012] Figure 1 shows an example of an ANN system to which the present invention is applied. FIG. 2 is a flowchart illustrating an example of nearest neighbor search within a cardinality-based vector database according to the present invention. FIG. 3 is a block diagram showing an example of a computing device according to the present invention. Specific details for implementing the invention

[0013] The terms used in this specification will be briefly explained, and an embodiment of the present invention will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in the present invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this specification should be defined not merely by their names, but based on their meanings and the overall content of the present invention.

[0014] The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification, and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of related prior art have been omitted where it is determined that such detailed descriptions could obscure the essence of the embodiments disclosed herein.

[0015] In the following description, when it is stated that a part is "connected (connected, in contact, or combined)" with another part, this includes not only cases where they are "directly connected (connected, in contact, or combined)" but also cases where they are "indirectly connected (connected, in contact, or combined)" with another member in between. Furthermore, when it is stated that a part "includes (provides or provides)" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may additionally "include (provides or provides)" other components.

[0016] The terms used herein are merely for describing specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise, and distributed components may be implemented in a combined form unless specifically limited. In this specification, terms such as "comprising" or "having" are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0017] Additionally, terms including ordinal numbers, such as first, second, etc., used in this specification may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0019] FIG. 1 shows an example of an ANN system (100) to which the present invention is applied.

[0020] Referring to FIG. 1, the ANN system may include at least one of a computing device (136), a document (140), a content repository (144), a user interface (148), a network (156), and an ANN search device (Approximate Nearest Neighbor Search device, 160).

[0021] A user utilizing the computing device (136) adds a document (140) to a content repository (144) and makes the document (140) searchable within the content repository (144). One or more areas, search terms, or keywords for designation included in the user interface (148) are provided to the user through the display of the computing device (136). That is, the user can enter keywords into the user interface (148), and the keywords are provided to the ANN search device (160) via the network (156). The ANN search device (160) finds the document most semantically similar to the keyword provided in the query and provides the document to the computing device (136).

[0022] For example, an ANN search device (160) can receive content from a content repository (144) and generate one or more vector indices for searching.

[0023] As another example, the ANN search device (160) can apply a deep learning model to a part of the content repository (144) and generate a vector index. Alternatively, an index (or vector index) regarding the ANN algorithm can be stored in advance and utilized.

[0024] As another example, if a user adds a document to the content repository (144), the ANN search device (160) can generate a vector representing the added document and add the generated vector to an existing index using vector addition.

[0025] As another example, if content is deleted or removed from an existing index, the ANN search device (160) can remove the corresponding vector or vector index.

[0026] As another example, an ANN search device can operate by generating a searchable dataset, applying a deep learning model to the generated dataset, and creating one or more vectors. The one or more vectors generated can represent a part of the dataset, and the dataset may include keywords, web pages, documents, images, videos, etc. Additionally, the deep learning model can generate a vector associated with each web page, document, image, or video; these vectors can each be indicated as nodes, and each node can indicate a list of the nearest nodes. Consequently, if a user wishes to query a dataset regarding the most similar content, the deep learning model generates a vector indicated by the query, and the vector index is searched using the generated vector.

[0028] Hereinafter, a nearest neighbor search method within a cardinality-based vector database according to the present invention is described. This method may be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. Additionally, this method may be performed by a gate or circuit associated with a processor, an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a System On Chip (SOC), or other hardware devices. In this method, a vector representing content to be added to a search index is received, wherein the received vector may be applied to the result of a deep learning model operating on the content, and the content to be added may be keywords, web pages, documents, images, etc.

[0029] FIG. 2 is a flowchart illustrating an example of nearest neighbor search within a cardinality-based vector database according to the present invention.

[0030] Referring to FIG. 2, cardinality data of a filter condition column used as a criterion for performance optimization during the filtering process is statistically recorded (S210).

[0031] Next, the statistical cardinality data recorded above is compared with a predetermined threshold value to determine and perform a data filtering and search method (S220).

[0032] For example, if the cardinality data is higher than a predetermined set value, general data filtering and an accurate k-nearest neighbor (KNN) search are performed on the result (S232). For example, the KNN search may be a search that classifies the data into a category containing more data among the neighbor data, and various disclosed methods to complement KNN may also be applied to the present invention.

[0033] As another example, if the cardinality data is lower than a predetermined set value, the search is not performed on neighbor nodes that do not satisfy the filtering condition during ANN search, and the search is performed by modifying the search algorithm so that the search space can be expanded by a multiple of the inverse value of the cardinality or filter combination probability (S234). For example, when using the HNSW algorithm, the search space is appropriately expanded during the greedy search process for the candidate set removed by filtering by increasing the value of ef-search by that multiple.

[0034] This is intended to compensate for the fact that accuracy decreases as the search space shrinks when searching for neighboring nodes that do not satisfy filtering conditions is not performed during ANN search.

[0036] Meanwhile, as an example according to the present invention, an embodiment for determining a predetermined threshold value in step S220 is as follows.

[0037] For example, when applying filtering, the cardinality data criterion can be set to a value where the number of cardinality unique values ​​is 25 or more.

[0038] As another example, when applying filtering, the cardinality data criterion can be determined as a setting value when the combined probability of the filter condition is 4% or less.

[0039] As the basis for determining this setting value, it can be noted that when performing a benchmark on the ANN index, the average qps performance based on 95% accuracy (or recall) shows performance that is about 30 times faster than the KNN method. In other words, since the performance gain is 30 times based on reasonable accuracy (e.g., 95%) for inaccurate results, the setting value according to the present invention is effective.

[0040] On the other hand, reducing the dataset itself to one-thirtieth of its original size through filtering alone yields the same level of performance; conversely, however, when applying inline filtering while traversing the ANN index, the performance difference is not significant, and the loss due to accuracy degradation is actually greater. In other words, while the ANN index itself offers the advantage of significantly improving performance on a log scale, performing additional searches using inline filters results in greater losses, rendering any performance gain unattainable.

[0041] According to the present invention, a new search method can be provided that solves the problem of search performance fluctuating according to filtering conditions and optimizes the integration of data filtering and ANN search to achieve higher performance and accuracy.

[0043] FIG. 3 is a block diagram showing an example of a computing device according to the present invention.

[0044] Referring to FIG. 3, the computing device (300) may include at least one processor (302) and system memory (304). Depending on the configuration and type of the computing device, the system memory (304) may be volatile storage, non-volatile storage, flash memory, or a combination of these memories.

[0045] The system memory (304) may include one or more program modules (306) suitable for running an operating system (305) and a software application (320), and is not limited to an approximate nearest neighbor (ANN) search device (323) and / or one or more components supported by the system described herein. For example, the ANN search device may receive content to be added, deleted, or searched. Additionally, the operating system (305) may be suitable for controlling the operation of the computing device (300).

[0046] Furthermore, embodiments of the present disclosure may be executed with a graphics library, another operating system, or any other application program and are not limited to any specific application or system.

[0047] The computing device (300) may have additional features or functions. For example, the computing device (300) may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tapes.

[0048] As mentioned above, a number of program modules and data files may be stored in system memory (304). While running on at least one processor (302), a program module (306) may perform a process that includes, but is not limited to, one or more aspects. Other program modules that may be used according to the present invention may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, picture or computer-aided application programs, etc.

[0049] Additionally, embodiments of the present invention may be implemented in individual electronic elements, a package or integrated electronic chip including logic gates, a circuit utilizing a microprocessor, or an electrical circuit including a single chip including electronic elements or a microprocessor. For example, embodiments of the present invention may be implemented through a system-on-a-chip (SOC) or integrated into a single integrated circuit, each or a number of the components of FIG. 3. Such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which may be integrated into a chip substrate as a single integrated circuit.

[0050] As used herein, the term "computer-readable medium" may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, or program modules. System memory (304), removable storage device (309), and non-removable storage device (310) are all examples of computer storage media, and computer storage media may include RAM, ROM, electrically eraseable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, DVD (digital multifunction disc) or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices, or any other manufactured product that may be used to store information and may be accessed by the computing device (300). Any such computer storage media may be part of the computing device (300).

[0051] The preferred embodiments of the present invention described above are disclosed for illustrative purposes only, and those skilled in the art with ordinary knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions should be considered to fall within the scope of the above claims.

[0052] Since various substitutions, modifications, and changes are possible within the scope of the technical concept of the present invention for those skilled in the art to which the present invention pertains, the present invention is not limited by the aforementioned embodiments and attached drawings.

Claims

Claim 1 A method for searching for the nearest neighbor within a cardinality-based vector database, comprising the step of statistically recording cardinality data including the number of cardinality eigenvalues ​​of a filter condition column or the joint probability of a filter condition, which is used as a criterion for optimizing search performance during the filtering process; A search method comprising the step of comparing the statistically recorded cardinality data with a predetermined threshold value and, based on the comparison result, selectively determining one of a first search method that performs an accurate k-nearest neighbor (KNN) search on the filtered data and a second search method that expands the search space without performing a search to neighbor nodes that do not satisfy the filtering condition during an ANN search, wherein the determination is characterized in that if the cardinality data is higher than the predetermined threshold value, the first search method is selected to perform data filtering and an accurate KNN search on the result, and if the cardinality data is lower than the predetermined threshold value, the second search method is selected to perform a search by modifying the search algorithm so as not to perform a search to neighbor nodes that do not satisfy the filtering condition during an ANN search, and to expand the search space by a multiple corresponding to either the inverse value of the joint probability of the filter condition or the number of cardinality eigenvalues. Claim 2 A search method according to claim 1, wherein the second search method is characterized by using the HNSW algorithm for the nearest neighbor search and increasing the value of ef-search by the multiple if the cardinality data is lower than a predetermined set value, thereby expanding the search space during the greedy search process for the candidate set removed by filtering. Claim 3 A search method according to claim 1, wherein the predetermined threshold value is determined based on 25 when the cardinality data is data regarding the number of cardinality eigenvalues, and based on 4% when the cardinality data is data regarding the combination probability of filter conditions. Claim 4 A computing device that performs a Nearest Neighbor search within a cardinality-based vector database comprises at least one processor; and system memory for executing operating systems and software applications; wherein the system memory statistically records cardinality data including the number of cardinality eigenvalues ​​of a filter condition column or the joint probability of a filter condition, which is used as a criterion for optimizing search performance in a filtering process; the processor compares the recorded statistically recorded cardinality data with a predetermined threshold value, and based on the comparison result, selectively determines one of a first search method that performs an accurate k-nearest neighbor (KNN) search on the filtered data and a second search method that expands the search space without performing a search to neighbor nodes that do not satisfy the filtering condition during an ANN search; wherein the decision determines that if the cardinality data is higher than the predetermined threshold value, the first search method is selected to perform data filtering and an accurate KNN search on the result; and if the cardinality data is lower than the predetermined threshold value, the second search method is selected to expand the search space without performing a search to neighbor nodes that do not satisfy the filtering condition during an ANN search, and the search space is of the joint probability of the filter condition A computing device characterized by determining to perform a search by modifying a search algorithm so as to be expandable to a multiple corresponding to either the inverse value or the number of cardinality eigenvalues. Claim 5 A computing device according to claim 4, wherein the second search method is characterized by using the HNSW algorithm for the nearest neighbor search and expanding the search space during the greedy search process for the candidate set removed by filtering by increasing the value of ef-search by the multiple. Claim 6 A computing device according to claim 4, wherein the system memory stores a setting value determined based on 25 when the cardinality data is data regarding the number of cardinality eigenvalues, and stores a setting value determined based on 4% when the cardinality data is data regarding the joint probability of filter conditions.

Citation Information

Patent Citations

  • Information processing device, information processing method, and information processing program

    JP2023015914A

  • Associative graph search

    KR1020220158209A

  • Approximate nearest neighbor search engine

    US20210406321A1