Database device, data retrieval method, and database system

Abstract

We provide a database device, data retrieval method, and database system that can achieve both accuracy in searching and speed in processing. [Solution] A database device comprising: a centroid determination module 513 that determines the centroid by clustering first vector data, which is vectorized data obtained by dividing a document into chunks, into multiple clusters; an index creation module 515 that creates a list of first vector data included in each cluster and predetermined areas that straddle the boundaries of each cluster; and a vector neighbor ranking module 517 that, when searching for first vector data close to second vector data created for a query, compares using first vector data included in the cluster to which the second vector data belongs, and first vector data included in a predetermined area set with respect to the boundary of the cluster to which the second vector data belongs.

Description

Technical Field

[0001] The present invention relates to a database device, a data search method, and a database system. In particular, the present invention relates to a database device used when searching for vector data.

Background Art

[0002] In recent years, with the development of large language models (LLMs), the demand for vector databases used in retrieval-augmented generation (RAG) has been increasing.

[0003] Patent Document 1 describes a clustering device that assigns a plurality of feature vectors respectively arranged on a spherical surface to a plurality of clusters by an iterative procedure. This clustering device has, for each feature vector, similarity calculation means for calculating the similarity between the feature vector and the representative vectors of each of the plurality of clusters, and determination means for determining whether or not to calculate the similarity at the t-th iteration. The determination means determines whether or not to calculate the similarity at the t-th iteration using at least the upper limit value calculated for the similarity based on the angle formed by the feature vector and the representative vector, and using at least the upper limit value at the (t - 1)-th iteration. The similarity calculation means does not calculate the similarity at the t-th iteration when it is determined by the determination means not to calculate the similarity.

[0004] Patent Document 2 describes a verification device for verifying the correspondence between two images. In this verification device, a feature extraction unit applies a convolutional neural network to each of the two images and obtains the output of the convolutional layer for each sub-region. A correspondence candidate calculation unit calculates the cosine similarity of the output of the convolutional layer for each combination of a sub-region of one image and a sub-region of the other image. If the cosine similarity is higher than a threshold, and the sub-region of the other image that has the greatest cosine similarity to sub-region A matches sub-region B, and the sub-region of one image that has the greatest cosine similarity to sub-region B matches sub-region A, then the combination of sub-region A of one image and sub-region B of the other image is selected as a correspondence candidate. The verification unit determines the appropriateness of each correspondence candidate based on the position coordinates of the sub-regions, and if the correspondence candidate is appropriate, it outputs the correspondence candidate as a correspondence. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2019-121044 [Patent Document 2] Japanese Patent Publication No. 2019-28700 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] A vector database works by retrieving a specified number of vectors that are close to the high-dimensional vector being queried, sorted by proximity. At this time, calculating the distance to all stored vectors would be computationally intensive. Therefore, an index is created in advance, and during the search, the calculation of the distance is narrowed down to vector data that is considered to be close to the query's vector data, thereby reducing the processing load and speeding up the process. However, with this method, if the accuracy of the search is prioritized, the processing becomes heavy, and the search speed tends to be slow. The present invention aims to provide a database device, a data retrieval method, and a database system that can achieve both the reliability of retrieval and the speed of processing. [Means for solving the problem]

[0007] To solve the above problems, the present invention provides a database device comprising: a clustering unit that divides first vector data, which is vectorized data obtained by dividing a document into chunks, into multiple clusters by clustering and determines the centroid of each cluster; a list creation unit that creates a list of first vector data belonging to each cluster and first vector data included in predetermined areas that straddle the boundaries of each cluster; and a search unit that, when searching for first vector data close to second vector data by comparing the first vector data with second vector data created for a query based on the list, uses first vector data included in the cluster to which second vector data belongs and first vector data included in predetermined areas set with respect to the boundaries of the cluster to which second vector data belongs for comparison. In this case, a database device that can achieve both search reliability and processing speed can be provided.

[0008] Here, for example, the list creation unit determines the first vector data included in a given region based on the position of the centroid and the position of the boundary. In this case, the first vector data near the boundary can be extracted. Furthermore, for example, the list creation unit determines the first vector data to be included in a given region based on the distance between the first vector data and the position of the centroid and the position of the boundary. In this case, the first vector data that is close to the boundary can be extracted based on the distance in the vector space. Furthermore, for example, the list creation unit determines the first vector data to be included in a given region based on the cosine similarity determined by the relationship between the first vector data and the position of the centroid. In this case, the first vector data near the boundary can be extracted based on the cosine similarity between the two centroids and the first vector data. Furthermore, for example, the search unit further compares all the first vector data contained in clusters adjacent to the cluster to which the second vector data belongs, determined by the position of the centroid. In this case, the accuracy of the search is improved. For example, the search unit divides the process into two cases: one where it uses all of the first vector data contained in clusters adjacent to the cluster to which the second vector data belongs, and another where it uses the first vector data contained within a predetermined area set for the cluster to which the second vector data belongs. In this case, even if the accuracy of the search is improved, the processing speed is less likely to decrease. Furthermore, for example, the search unit uses all of the first vector data contained in a predetermined number of clusters adjacent to the cluster to which the second vector data belongs, in order of proximity of their centroids, and uses the first vector data contained in a predetermined area set for the cluster to which the second vector data belongs for the remaining clusters. In this case, the reliability of the search is improved by using all of the first vector data in clusters where there is a higher probability that a first vector data close to the second vector data exists, and the processing speed is improved by searching only the predetermined area in clusters where there is a lower probability that a first vector data close to the second vector data exists. Furthermore, for example, the list creation unit creates a first table representing the relationship between centroids or clusters, indicating whether or not the first vector data exists in a predetermined region. The search unit then refers to this first table to determine whether or not to search for the first vector data included in the predetermined region. In this case, it becomes easier to determine whether or not to search the predetermined region. Furthermore, for example, the list creation unit creates a second table showing a list of first vector data included in a predetermined area, representing the relationships between centroids or clusters, and the search unit identifies the list of first vector data included in the predetermined area by referring to the second table. In this case, the location of the first vector data included in the predetermined area can be determined. For example, the search unit creates a third table indicating whether a predetermined area was used during the search, representing the relationship between centroids or clusters. By referring to this third table, it avoids duplicate comparisons of the predetermined area. In this case, the processing speed during the search is improved.

[0009] Furthermore, the present invention provides a data retrieval method in which a processor executes a program recorded in memory to cluster first vector data, which is vectorized data obtained by dividing a document into chunks, into multiple clusters, determines the centroid of each cluster, creates a list of first vector data belonging to each cluster and first vector data included in predetermined areas that straddle the boundaries of each cluster, and compares the first vector data with the second vector data created for the query based on the list to search for first vector data that is close to second vector data. In this case, a data retrieval method can be provided that can achieve both the reliability of the search and the speed of processing.

[0010] Furthermore, the present invention provides a database system comprising: a terminal device for a client to input a query; a document management device for dividing a document into chunks; and a database device for searching for chunks close to the query. The database device includes: a clustering unit that divides first vector data, which is vectorized chunk data, into multiple clusters by clustering and determines the centroid of each cluster; a list creation unit that creates a list of first vector data belonging to each cluster and first vector data included in predetermined areas that straddle the boundaries of each cluster; and a search unit that, when searching for first vector data close to second vector data by comparing the first vector data with second vector data created for the query based on the list, uses first vector data included in the cluster to which second vector data belongs and first vector data included in predetermined areas set with respect to the boundaries of the cluster to which second vector data belongs for comparison. In this case, a database system that can achieve both search accuracy and processing speed can be provided. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide a database device, a data retrieval method, and a database system that can achieve both the reliability of retrieval and the speed of processing. [Brief explanation of the drawing]

[0012] [Figure 1] This figure illustrates the schematic operation of the database system of this embodiment. [Figure 2] Figures (a) to (c) illustrate the operation of index calculation using the conventional search space narrowing method (Ivfflat). [Figure 3] (a) is a diagram illustrating a case where selection is missed when only the closest centroid point is selected. (b) is a diagram illustrating a method for creating a cluster to which this embodiment is applied. [Figure 4]This is a diagram explaining a method for selecting vector data that enters a predetermined area. [Figure 5] This is a diagram showing the hardware configuration and functional configuration of a vector database server. [Figure 6] This is a diagram showing the data structure of a data page used in this embodiment. [Figure 7] This is a diagram showing the data structure of an adjacent centroid table. [Figure 8] This is a diagram showing the data structure of an adjacent centroid page table. [Figure 9] This is a diagram showing the data structure of an adjacent centroid usage flag table. [Figure 10] This is a flowchart explaining the operation of an import execution module of a vector database server. [Figure 11] This is a flowchart explaining the operation of a query execution module of a vector database server. [Figure 12] This is a diagram showing the processing when mixing the case of fully searching for adjacent clusters and the case of searching only a predetermined area where the boundary K with an adjacent cluster is set. [Figure 13] This is a diagram showing a method for selecting first vector data that enters a predetermined area based on the cosine similarity determined by the relationship between the first vector data and the position of the centroid. [Figure 14] This is a flowchart explaining the operation of an import execution module in the second embodiment.

Embodiment for Carrying Out the Invention

[0013] Hereinafter, referring to the accompanying drawings, embodiments of the present invention will be described in detail. Hereinafter, the present invention will be described by the first embodiment to the second embodiment.

[0014] [First Embodiment] <Explanation of the Schematic Operation of Database System 100> Figure 1 is a diagram illustrating the schematic operation of the database system 100 in this embodiment. The illustrated database system 100 is a device in which a client 502 inputs a query and outputs a response to that query. Document DB (database) 101 is an example of a document management system that divides documents into chunks. Document DB 101 retrieves document data from the archive and stores the documents in chunks. Documents are not particularly limited, but examples include specifications. A chunk is a unit into which document data is divided to make it easier to handle, and it divides a document into units such as sentences, paragraphs, words, and phrases. The vector database 102 performs embedding for each chunk to generate and store vector data. Embedding is the operation of converting a chunk into a vector representation. This converts the chunk into vector data.

[0015] Meanwhile, client 502, which uses the database system 100, enters queries using its own terminal device. A query is a keyword or phrase used to retrieve information. Queries sent to the database system 100 are embedded and converted into vector representations. In this way, the queries are converted into vector data.

[0016] Vector DB102 is an example of a database device that searches for chunks similar to a query. Vector DB102 compares the vector data generated from the query with the vector data generated from the chunks obtained by dividing the document, and searches for vector data from the latter that is similar to the former. In this case, by searching for vector data that are close in distance in the vector space, chunks that are similar in meaning to the query are found. A predetermined number of vector data found in order of proximity are sent to Document DB101 as a candidate list. Document DB101 retrieves the chunks corresponding to the candidate list from Document DB101 and outputs them to LLM103 as candidate documents (chunks). LLM103 generates a response to the query based on candidate documents (chunks) and sends it to client 502.

[0017] Calculating the distances of all stored vector data would be computationally intensive. Therefore, an index is created in advance for the vector data generated from the chunks obtained by dividing the document, and during the search, an index calculation is performed to narrow down the search to vector data that is considered close to the query vector data, and calculate the distance based on the index. In this embodiment, the search space narrowing method (Ivfflat) is used for the index calculation. From now on, vector data generated from the chunks obtained by dividing the document may be referred to as "first vector data." Similarly, vector data generated from queries may be referred to as "second vector data."

[0018] <Explanation of Index Calculation> Figures 2(a) to 2(c) illustrate the operation of index calculation using the conventional search space narrowing method (Ivfflat). Figure 2(a) shows that there are many first vector data points, indicated by white circles, in the vector space. This vector space is multidimensional, for example, 128-dimensional. Figure 2(b) illustrates the process of index creation. Here, the first vector data is divided into clusters indicated by dotted lines using the k-means clustering method, one of the clustering techniques. The centroids of these clusters are registered as indices, and a list of the first vector data belonging to the cluster containing the centroid is created. Figure 2(b) shows that the first vector data is divided into clusters shown as cluster1 and cluster2, with their respective centroids being centroid1 and centroid2.

[0019] Figure 2(c) illustrates the process of retrieving the first vector data using an index. Here, the second vector data generated from the query is shown as `query`. The distance between the second vector data and its centroid is calculated, and the L closest centroids are selected. The distance between all the first vector data and the second vector data within the cluster to which each centroid belongs is calculated, and the N closest first vector data are returned as results. Here, the first vector data with the closest distance is shown as `top1`.

[0020] Index calculation using the search space narrowing method (Ivfflat) can almost certainly find vector data that are close in distance, but it tends to be slow. On the other hand, in index calculation using the search space narrowing method (Ivfflat), the processing speed changes depending on the number of centroids selected. For example, if only the closest centroid is selected, the amount of data in the first vector used to calculate the distance can be narrowed, thus reducing the computational load. However, if the second vector data is located at the cluster boundary, omissions (close but not selected) are likely to occur. In contrast, if, for example, multiple (e.g., L) centroids are selected based on their proximity, the likelihood of data loss decreases, even if the second vector data is located at the cluster boundary. However, the processing load increases because the distances to all first vector data within the cluster to which the L centroids belong are calculated. In other words, the fewer items you select from those with similar centroids, the more likely omissions are to occur, but the less computational processing required. Conversely, the more items you select, the less likely omissions are to occur, but the more computational processing is required. In short, there is a trade-off between the two. Therefore, it is necessary to create clusters that minimize leakage even with a small number of centroids.

[0021] Figure 3(a) illustrates a case where selection is missed when only the closest centroid point is selected. Here, we show cluster 0, to which centroid 0 belongs, and cluster 1, to which centroid 1 belongs. For the sake of clarity, clusters 0 and 1 are illustrated as pentagons in a two-dimensional plane. Clusters 0 and 1 are adjacent, with a boundary K between them. The number following the centroid is the number assigned to the centroid, and will hereafter be referred to as the "centroid vector number."

[0022] If the second vector data generated from the query is query point 1, then all first vector data in cluster 0 are searched, while the first vector data in cluster 1 is not searched. As a result, target point 1 is selected as the closest first vector data, and in this case, no problem occurs. In contrast, if the second vector data is query point 2, which is close to the boundary K between cluster 0 and cluster 1, then all first vector data within cluster 0 will be searched, while the first vector data within cluster 1 will not be searched. In this case, the first vector data closest to query point 2 is target point 2, but target point 2 is not selected because it is within cluster 1. In other words, target point 2 is not selected, resulting in an oversight.

[0023] Figure 3(b) shows a diagram illustrating how to create a cluster to which this embodiment is applied. In this embodiment, a predetermined region R that straddles the boundary K between cluster 0 and cluster 1 is defined as the overlapping portion of cluster 0 and cluster 1. For the sake of clarity, the predetermined region R is shown here as a rectangle in a two-dimensional plane. In this case, target point 2, which is the first vector data closest to query point 2, belongs to the predetermined region R. When all first vector data within cluster 0 is searched, the predetermined region R, which is the overlapping portion, is also included in the search. As a result, target point 2 is selected, and no data is missed.

[0024] <Explanation of the method for selecting the first vector data to fall within the predetermined region R> Figure 4 illustrates the method for selecting the first vector data to fall within a predetermined region R. In Figure 4, the boundary K between cluster 0 and cluster 1 is shown as a boundary hyperplane. Furthermore, centroid point 0 and centroid point 1 are shown as the centroids of cluster 0 and cluster 1, respectively. Here, a ranking of the first vector data is created in ascending order of distance from the boundary hyperplane. If the number exceeds a certain number (X), the top X first vector data are assigned to a predetermined region R. The remaining vector data are assigned to the cluster of nearest centroids. X can be determined by calculating the average number of vectors belonging to a single centroid and taking n% of that average. In this case, n is adjustable. By adjusting n, the number of first vectors that enter the predetermined region R can be adjusted. Figure 4 shows the case where contour lines of distance from the boundary hyperplane, which indicate points with the same distance from the boundary hyperplane, are assigned to the predetermined region R as data points within the ranking, and the remaining first vector data 404 is treated as a data point outside the ranking. Furthermore, considering the distance from the centroid, if the first vector data is closer to the centroid than the boundary hyperplane, it can be excluded.

[0025] <Description of the functional configuration of Vector Database Server 501> Next, we will describe the functional configuration of the vector database server 501 that realizes the above configuration. Figure 5 shows the hardware and functional configuration of the vector database server 501. The vector database server 501 corresponds to the vector DB 102 in Figure 1. The vector database server 501 can be, for example, a cloud server used when providing the database system 100 service on the cloud.

[0026] The vector database server 501 is a computer device comprising a CPU (Central Processing Unit) 503, memory 504, and storage 505. The CPU503 is an example of a processor, a means of calculation, and it executes various software such as the OS (operating system) and application programs. Memory 504 is a memory area that stores various software and the data used to run it. Storage 505 is a memory area that stores input data for various software and output data from various software. Storage 505 stores data pages 512, document data, and other various data, which will be described in more detail later.

[0027] Memory 504 stores application programs for implementing the functions of the vector database controller 506, import execution module 507, query execution module 508, vector distance module 509, vector index management data buffer 510, and page read / write module 511. The CPU 503 executes these application programs to implement each of these functions.

[0028] The vector database controller 506 controls the entire vector database server 501. The vector database controller 506 also accepts queries from the client 502. The import execution module 507 clusters the first vector data generated from the chunks obtained by dividing the document, registers the centroids as indices, and creates a list of vector data belonging to the cluster where the centroid is located. The query execution module 508 uses the created list to search for a first vector data that is close to the second vector data generated from the query, and then creates a list of a predetermined number of candidate first vector data in order of proximity. The vector distance module 509 calculates the distance between the first vector data and the second vector data. The query execution module 508 uses this distance to search for the first vector data that is closest to the second vector data. The vector index management data buffer 510 creates and modifies data pages 512. The page read / write module 511 reads and writes data pages 512 from the storage 505.

[0029] The import execution module 507 clusters the first vector data and creates a list of the first vector data. In other words, the import execution module 507 performs preprocessing before executing the processing for the query. This can also be described as the process of building a database of vector data that will answer the query. The import execution module 507 comprises a centroid determination module 513, a centroid / target point boundary determination module 514, and an index creation module 515.

[0030] The centroid determination module 513 is an example of a clustering unit. It divides the vector data (first vector data), which is data obtained by vectorizing the chunks obtained by dividing the document, into multiple clusters and determines the centroid of each cluster. The centroid / target point boundary determination module 514 extracts the first vector data included in the predetermined region R as described in Figures 3 and 4. The index creation module 515 registers the centroid as an index. The index creation module 515 is also an example of a list creation unit, and it creates a list of the first vector data belonging to each cluster and the first vector data included in a predetermined region R that straddles the boundary K of each cluster.

[0031] The query execution module 508 compares the first vector data with the second vector data and searches for the first vector data that is closest to the second vector data. This can also be described as the processing performed when responding to a query. The query execution module 508 includes a query vector holding module 516 and a vector neighbor ranking module 517. The query vector storage module 516 stores the second vector data created for the query. The vector neighbor ranking module 517 is an example of a search unit. Based on the list created by the index creation module 515, it compares the first vector data with the second vector data created for the query. When searching for the first vector data that is close to the second vector data, it uses the first vector data included in the cluster to which the second vector data belongs, and the first vector data included in a predetermined region R set with respect to the boundary K of the cluster to which the second vector data belongs, for comparison.

[0032] <Explanation of the data structure of data page 512> Figure 6 shows the data structure of data page 512 used in this embodiment. In this embodiment, each data point is associated with the others in a tree structure. The first layer of the tree structure is the centroid vector data 601, which stores the vector data of the centroid of each cluster. Here, we show that it stores the centroid 0 vector 602, which is the vector data of centroid point 0, the centroid 1 vector 603, and so on, which is the vector data of centroid point 2.

[0033] The second layer consists of three types of table data. These tables are the adjacent centroid table 604, the adjacent centroid page table 605, and the adjacent centroid usage flag table 606. More details about these tables will be provided later.

[0034] The third layer is a page list associated with the adjacent centroid page table 605. This page list stores the first vector data contained in each cluster to which each centroid belongs, and the first vector data contained in a predetermined region R set across the boundary K of the clusters to which each centroid belongs. Here, the page list for centroid point 0 is the centroid 0 page list 607, the page list for the predetermined region R that straddles the boundary K of the clusters for centroid point 0 and centroid point 1 is the centroid 0 & 1 overlap page list 610, and the page list for centroid point 1 is the centroid 1 page list 613.

[0035] Each page list consists of multiple pages, as shown in the fourth layer. Here, we show that the centroid 0 page list 607 consists of centroid 0 page 1, centroid 0 page 2, ... Also, we show that the centroid 0&1 duplicate page list 610 consists of centroid 0&1 page 1, centroid 0&1 page 2, ... Furthermore, we show that the centroid 1 page list 613 consists of centroid 1 page 1, centroid 1 page 2, ...

[0036] Figure 7 shows the data structure of the adjacent centroid table 604. The adjacent centroid table 604 is an example of a first table that indicates whether or not the first vector data exists in a given region R. Here, the adjacent centroid table 604 is created as a relationship between two centroids. When the flag written at the intersection of the two centroids is "0", it indicates that the clusters to which the respective centroids belong are not adjacent, or that the first vector data does not exist in the given region R set with respect to the boundary K of the clusters to which these centroids belong. Conversely, when the flag written at the intersection of the two centroids is "1", it indicates that the first vector data exists in this given region R. Here, the adjacent centroid table 604 is created as a relationship between two centroids, but it may also be created as a relationship between two clusters.

[0037] By referring to the adjacent centroid table 604, it is possible to determine whether or not to add the first vector data included in the predetermined region R to the search. For example, if the second vector data is included in cluster 0 to which centroid 0 belongs, the data for centroid 1 relative to centroid 0 is "1". Therefore, the predetermined region R is set between these centroids (between cluster 0 and cluster 1), and the first vector data exists. For this reason, a search is necessary not only for cluster 0 but also for this location. In contrast, the data for centroid 2 relative to centroid 0 is "0". Therefore, the predetermined region R is not set between these centroids (between cluster 0 and cluster 2), and the first vector data does not exist. For this reason, a search is not necessary for this location.

[0038] Figure 8 shows the data structure of the adjacent centroid page table 605. The adjacent centroid page table 605 is an example of a second table that shows a list of first vector data contained in a given region R. Here, the adjacent centroid page table 605 is created as a relationship between two centroids, and the page list between these centroids is shown. For example, the page list of centroid 0 and the intersection of centroid 0 is the page list for centroid 0, and is the centroid 0 page list 607 in Figure 6. Also, the page list of centroid 0 and centroid 1 is the centroid 0&1 overlap page list 610. Furthermore, the page list of centroid 1 and centroid 1 is the page list for centroid 1, and is the centroid 1 page list 613. By referring to the adjacent centroid page table 605, the page list of first vector data contained in a given region R can be identified. That is, the adjacent centroid page table 605 allows us to know the location of first vector data contained in a given region. In this example, the adjacent centroid page table 605 was created to represent the relationship between two centroids, but it could also be created to represent the relationship between two clusters.

[0039] Figure 9 shows the data structure of the adjacent centroid usage flag table 606. The adjacent centroid usage flag table 606 is an example of a third table that indicates whether or not a predetermined region R was used during a search. Here, the adjacent centroid usage flag table 606 is created as a relationship between two centroids. An adjacent centroid flag is recorded at the intersection of the two centroids. If this adjacent centroid flag is "0", it indicates that the predetermined region R set for the cluster boundary K to which these centroids belong has not yet been used in the search. Conversely, if this adjacent centroid flag is "1", it indicates that this predetermined region R has already been used in the search. Note that here, the adjacent centroid usage flag table 606 is created as a relationship between two centroids, but it may also be created as a relationship between two clusters.

[0040] By referring to the adjacent centroid usage flag table 606, it is possible to avoid duplicate comparisons for a given region R. This improves the processing speed during searches. For example, if the second vector data is included in cluster 0, to which centroid 0 belongs, and the centroid 0&1 duplicate page list 610 is searched, the data for centroid 1 relative to centroid 0 changes from "0" to "1". This indicates that the search for the given region R between cluster 0 and cluster 1 has already been completed. Subsequently, when cluster 1 becomes the target of the search, there is no need to search the given region R between cluster 0 and cluster 1 because the search has already been completed. Therefore, by referring to the fact that the data for centroid 1 relative to centroid 0 is "1", it is clear that there is no need to search, and duplicate comparisons for the given region R can be avoided.

[0041] <Description of the operation of Vector Database Server 501> Next, we will describe the operation of the vector database server 501. Figure 10 is a flowchart illustrating the operation of the import execution module 507 of the vector database server 501. Specifically, Figure 10 illustrates the preprocessing steps performed before executing the processing of a query.

[0042] First, the import execution module 507 samples and obtains a first vector data from the overall vector data to be used to determine the centroid (S1001). Next, the import execution module 507 performs clustering using the k-means method with the sampled first vector data (S1002). Then, the import execution module 507 saves a predetermined number of centroid coordinates to the header page (S1003).

[0043] Next, the import execution module 507 stores the centroid vector numbers of each centroid or two centroids in the adjacent centroid page table 605 (S1004). Furthermore, the import execution module 507 retrieves one first vector data (target point) to create an index (S1005). Then, the import execution module 507 calculates the distance to each centroid for the extracted first vector data and extracts the nearest centroid and the second nearest centroid (S1006).

[0044] Next, the import execution module 507 determines whether the distance from the boundary hyperplane to the extracted first vector data is less than or equal to the distance from the centroid to the boundary hyperplane (S1007). As a result, if the distance from the boundary hyperplane is less than or equal to the distance from the centroid to the boundary hyperplane (Yes in S1007), the import execution module 507 determines whether this distance is shorter than the maximum value in the distance ranking list from the boundary hyperplane (S1008). Then, if this distance is shorter than the maximum in the ranking list of distances from the plane (Yes in S1008), the import execution module 507 removes the maximum from the current distance ranking. Then, it adds the target point to the page list of the nearest centroid (S1009).

[0045] Then, the import execution module 507 registers the target point in the distance ranking list from the boundary hyperplane of the page list of the two centroids (S1010). Next, the import execution module 507 sets the intersection of the nearest centroid and the second nearest centroid in the adjacent centroid table 604 to 1 (S1011). Furthermore, the import execution module 507 determines whether the vector data obtained in S1001 is the last vector data (S1012). Then, if it was the last vector data (Yes in S1012), the import execution module 507 adds the ranking list to the page list of the two centroids (S1013). Conversely, if it is not the last vector data (No in S1012), the process returns to S1005.

[0046] Furthermore, in S1007, if the distance from the boundary hyperplane exceeds the boundary hyperplane from the centroid (No in S1007), and in S1008, if the distance is the same as or longer than the maximum value in the ranking list of distances from the plane (No in S1008), the index will only include the centroid closest to the target point. Therefore, the import execution module 507 adds the target point to the page list of the closest centroid (S1014). After that, it proceeds to S1012.

[0047] In the processing steps S1007-S1010, the import execution module 507 determines the first vector data to be included in a given region R based on the position of the centroid and the position of boundary K. This allows for the extraction of the first vector data that is close to boundary K. More specifically, the first vector data to be included in a given region R is determined by the distance between the first vector data and the positions of the centroid and boundary K. This allows for the extraction of the first vector data that is close to boundary K based on distance in vector space. This processing is performed by the index creation module 515.

[0048] Figure 11 is a flowchart illustrating the operation of the query execution module 508 of the vector database server 501. Specifically, Figure 11 illustrates the process of searching for the first vector data that is closest to the second vector data when responding to a query.

[0049] First, the query execution module 508 obtains a second vector data, which is the vector data of the query (S1101). Next, the query execution module 508 compares the second vector data with all the centroid vector data and obtains a list of the top n centroid vector data (S1102). Furthermore, the query execution module 508 sequentially retrieves one centroid vector number from the list of centroid vector data (S1103).

[0050] Then, the query execution module 508 obtains the page number corresponding to the centroid vector number retrieved in S1103 from the adjacent centroid table 604 and adds it to the search page list (S1104). Next, the query execution module 508 sequentially retrieves from the adjacent centroid table 604 the numbers in which data is present in both of the extracted centroid vector numbers (S1105). Furthermore, the query execution module 508 obtains adjacent centroid flags corresponding to numbers where data is present in both of the obtained centroid vector numbers (S1106).

[0051] Then, the query execution module 508 determines whether the acquired adjacent centroid flag is 0 or not (S1107). As a result, if the acquired adjacent centroid flag is 0 (Yes in S1107), the query execution module 508 sets the adjacent centroid flag corresponding to the number in which data is present in both of the acquired centroid vector numbers to 1 (S1108). If the acquired adjacent centroid flag is not 0 (it is 1) (No in S1107), the process proceeds to S1110. Furthermore, the query execution module 508 obtains the page number corresponding to the number in which data is entered in both centroid vector numbers and adds it to the search page list (S1109).

[0052] Next, the query execution module 508 determines whether the last two numbers contain data (S1110). As a result, if the last number contains data in both fields (Yes in S1110), the query execution module 508 determines whether it is the last centroid vector number (S1111). If the last number does not contain data in both fields (No in S1110), the process returns to S1105. As a result, if it is the last centroid vector number (Yes in S1111), the query execution module 508 retrieves one element at a time from the resulting search page list (S1112). If it is not the centroid vector number (No in S1111), the process returns to S1103.

[0053] Next, the query execution module 508 obtains the first vector, which is included in the page list written in the extracted element, compares it with the second vector data, creates a ranking, and retains a predetermined quantity (S1113). Furthermore, the query execution module 508 merges the previously obtained ranking with the currently obtained ranking and retains a predetermined quantity (S1114). Then, the query execution module 508 determines whether it is the last element in the search page list (1115). As a result, if it is the last step (Yes in S1115), the query execution module 508 terminates the series of operations. If it is not the last step (No in S1115), it returns to S1112.

[0054] Furthermore, it is possible to combine a full search of adjacent clusters with a search of only a predetermined region R defined by the boundary K between adjacent clusters. This can also be described as the query execution module 508 further using and comparing all first vector data contained in clusters adjacent to the cluster to which the second vector data belongs, which is determined by the position of the centroid. This improves the certainty of the search. Alternatively, the query execution module 508 can be described as dividing the process into two cases: using all first vector data contained in clusters adjacent to the cluster to which the second vector data belongs, and using first vector data contained in a predetermined region R set for the cluster to which the second vector data belongs. This prevents a decrease in processing speed even when the certainty of the search is improved.

[0055] Figure 12 illustrates the process when a combination of searching all adjacent clusters and searching only a predetermined region R defined by the boundary K between adjacent clusters is performed. In this case, the query execution module 508 uses all of the first vector data contained in a predetermined number of clusters adjacent to the cluster to which the second vector data belongs, in order of proximity of their centroids, and for the remaining clusters, it uses the first vector data contained in a predetermined region R set for the cluster to which the second vector data belongs. This improves the certainty of the search by using all of the first vector data in clusters where there is a higher probability that a first vector data close to the second vector data exists, and improves the processing speed by searching only the predetermined region R in clusters where there is a lower probability that a first vector data close to the second vector data exists.

[0056] Figure 12 shows that the centroid 0 where the query point exists has a large number of neighboring clusters. For example, in 128 dimensions, even if there are two neighboring clusters per dimension, there are 256 neighboring clusters. In the current index calculation using the search space narrowing method (Ivfflat), at most 50 neighboring clusters are selected from the closest ones for the search. In other words, the remaining 206 neighboring clusters are discarded, which reduces recall (search accuracy). To avoid this, for example, for the 50 neighboring clusters (the 50 clusters in order of proximity of the centroid), all the first vector data contained within them are searched (full search). On the other hand, for the remaining 206 neighboring clusters (the remaining clusters), the first vector data contained within a given region R is searched, and other first vector data contained within this cluster are not included in the search. This increases recall and reduces the number of adjacent clusters to select. There is a trade-off between the number of clusters to perform a full search on and processing speed. Therefore, increasing the number of clusters to perform a full search on improves recall but decreases processing speed. Conversely, decreasing the number of clusters to perform a full search on improves recall but increases processing speed. Thus, the number of clusters to perform a full search on can be adjusted according to the required processing speed and computing power.

[0057] [Second Embodiment] In the first embodiment, the import execution module 507 determined the first vector data included in a predetermined region R based on the position of the centroid and the position of the boundary K. However, it is also possible to determine the first vector data included in a predetermined region R by the cosine similarity determined by the relationship between the first vector data and the position of the centroid. In this case, the cosine similarity allows for the extraction of first vector data near the boundary K.

[0058] Figure 13 illustrates a method for selecting a first vector data point that falls within a predetermined region R, based on the cosine similarity determined by the relationship between the first vector data point and the position of its centroid. Figure 13 illustrates the boundary hyperplane as the boundary K between cluster 0 and cluster 1. It also illustrates centroid points 0 and 1 as the centroids of cluster 0 and cluster 1, respectively.

[0059] Here, we consider the cosine similarity (i.e., angle) of two vectors 1305 and 1306, which are formed by connecting the nearest neighbor centroid point and the second nearest neighbor centroid point from the first vector data 1301. If the cosine similarity value is less than or equal to a certain value (i.e., it lies between the two centroids), then the vectors 1305 and 1306 are assigned to a predetermined region R if the absolute value of the difference in their vector lengths is less than or equal to a certain value. Otherwise, they are assigned to the cluster of nearest neighbor centroid points. By adjusting the constant value set for the cosine similarity value, the cosine similarity contour line 1307 in Figure 13 changes as shown by contour lines 1308 and 1309, allowing the predetermined region R to be adjusted vertically in the figure. Similarly, by adjusting the constant value set for the cosine similarity value, the contour line 1310 of the absolute difference in vector lengths in Figure 13 changes, allowing the predetermined region R to be adjusted horizontally in the figure. Figure 13 shows the case where the top three (1st to 3rd) first vector data points 1301 to 1303, which have the smallest distance from the boundary hyperplane, are assigned to a predetermined region R, while the other first vector data point 1304 is excluded from the ranking.

[0060] Furthermore, for the first vector data, a ranking can be created from the smallest to the largest based on the absolute difference in vector lengths. If the number of such differences exceeds a certain number (X), only the top X vectors are assigned to a predetermined region R, while the others are assigned to the cluster of the nearest centroid. X can be determined by calculating the average number of vectors belonging to a single centroid and taking n% of that average. In this case, n is adjustable. By adjusting n, the number of first vectors that enter the predetermined region R can be directly adjusted.

[0061] This method improves the processing speed when determining the first vector data included in a predetermined region R compared to the first embodiment. On the other hand, the accuracy of the search is less favorable compared to the first embodiment, as there is a higher risk of omissions. In other words, this method selects data within the range indicated by the cosine similarity contour line 1307. As mentioned above, this range can be changed by changing a certain value, as shown by contour lines 1308 and 1309, but the upper and lower parts of the figure are difficult to cover, resulting in a higher risk of omissions.

[0062] Figure 14 is a flowchart illustrating the operation of the import execution module 507 in the second embodiment. S1401-S1406 and S1409-S1415 in Figure 14 are the same as S1001-S1006 and S1008-S1014 in Figure 10, with only S1407-S1408 being different. Therefore, the following explanation will focus on S1407-S1408, which differs from those in Figure 10. The import execution module 507 calculates the dot product of the vector of the centroid closest to the target point and the vector of the centroid second closest to the target point (S1407). Then, the import execution module 507 determines whether the cosine similarity < the reference value and the absolute value of the difference in vector lengths < the reference value (S1408). As a result, if the condition of S1408 is met (Yes in S1408), proceed to S1409; otherwise, proceed to S1415.

[0063] As described above, the vector database server 501 and database system 100 make it possible to achieve both search accuracy and processing speed.

[0064] <Explanation of data search methods> The processing performed by the vector database server 501 is realized through the cooperation of software and hardware resources. Specifically, a processor such as the CPU 503 installed in the vector database server 501 loads programs that realize each function of the vector database server 501 into memory 504 and executes them, thereby realizing each of these functions. Therefore, the processing performed by the vector database server 501 described above can be understood as a data retrieval method in which, by executing a program stored in memory, the processor clusters the first vector data, which is vectorized data obtained by dividing a document into chunks, into multiple clusters, determines the centroid of each cluster, creates a list of the first vector data belonging to each cluster and the first vector data contained in a predetermined region R that straddles the boundary of each cluster, and compares the first vector data with the second vector data created for the query based on the list, and when searching for first vector data that is close to the second vector data, the first vector data contained in the cluster to which the second vector data belongs and the first vector data contained in a predetermined region R set with respect to the boundary of the cluster to which the second vector data belongs are used for comparison. This makes it possible to provide a data retrieval method that can achieve both the reliability of the search and the speed of processing.

[0065] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are explained in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations. Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. Alternatively, each of the above configurations and functions may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, and files that implement each function can be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In reality, it is safe to assume that almost all components are interconnected. [Explanation of symbols]

[0066] 100…Database System, 501…Vector Database Server, 503…CPU, 504…Memory, 505…Storage, 506…Vector Database Controller, 507…Import Execution Module, 508…Query Execution Module, 512…Data Page, 513…Centroid Determination Module, 514…Centroid / Target Point Boundary Determination Module, 515…Index Creation Module, 516…Query Vector Holding Module, 517…Vector Neighbor Ranking Module, 601…Centroid Vector Data, 604…Adjacent Centroid Table, 605…Adjacent Centroid Page Table, 606…Adjacent Centroid Usage Flag Table

Claims

1. The first vector data, which is data obtained by vectorizing the chunks obtained by dividing the document, is divided into multiple clusters by clustering, and the clustering unit determines the centroid of each cluster. A list creation unit creates a list of first vector data belonging to each cluster and first vector data included in a predetermined region that straddles the boundary of each cluster, A search unit compares the first vector data with the second vector data created for the query based on the aforementioned list, and when searching for the first vector data that is close to the second vector data, it compares using the first vector data included in the cluster to which the second vector data belongs, and the first vector data included in the predetermined area set with respect to the boundary of the cluster to which the second vector data belongs. A database device equipped with the following features.

2. The database device according to claim 1, wherein the list creation unit determines the first vector data included in the predetermined region based on the position of the centroid and the position of the boundary.

3. The database device according to claim 2, wherein the list creation unit determines the first vector data to be included in the predetermined region based on the distance between the first vector data and the position of the centroid and the position of the boundary.

4. The database device according to claim 2, wherein the list creation unit determines the first vector data to be included in the predetermined region based on the cosine similarity determined by the relationship between the first vector data and the position of the centroid.

5. The database apparatus according to claim 1, wherein the search unit is determined by the position of the centroid and further compares all first vector data contained in clusters adjacent to the cluster to which the second vector data belongs.

6. The database device according to claim 5, wherein the search unit divides the search into two cases: one in which it uses all of the first vector data contained in a cluster adjacent to the cluster to which the second vector data belongs, and the other in which it uses the first vector data contained in the predetermined area set for the cluster to which the second vector data belongs.

7. The database device according to claim 6, wherein the search unit uses all of the first vector data contained in a predetermined number of clusters adjacent to the cluster to which the second vector data belongs, in order of proximity of the centroid position, and uses the first vector data contained in the predetermined area set for the cluster to which the second vector data belongs for the remaining clusters.

8. The list creation unit creates a first table representing whether or not a first vector data exists in the predetermined area, as a relationship between the centroids or between the clusters. The database device according to claim 1, wherein the search unit determines whether or not to search for the first vector data included in the predetermined area by referring to the first table.

9. The list creation unit creates a second table showing a list of first vector data included in the predetermined area, as a relationship between the centroids or between the clusters. The database device according to claim 1, wherein the search unit identifies a list of first vector data included in the predetermined area by referring to the second table.

10. The database device according to claim 1, wherein the search unit creates a third table indicating whether or not the predetermined area was used during the search, as a relationship between the centroids or between the clusters, and by referring to the third table, it avoids performing duplicate comparisons for the predetermined area.

11. The processor executes the program stored in memory, The first vector data, which is data obtained by vectorizing the chunks obtained by dividing the document, is divided into multiple clusters by clustering, and the centroid of each cluster is determined. Create a list of the first vector data belonging to each cluster and the first vector data included in a predetermined region that straddles the boundary of each cluster. When searching for a first vector data that is close to the second vector data by comparing the first vector data with the second vector data created for the query based on the aforementioned list, the comparison is made using the first vector data included in the cluster to which the second vector data belongs, and the first vector data included in the predetermined area set with respect to the boundary of the cluster to which the second vector data belongs. Data search methods.

12. A terminal device in which the client enters queries, A document management system that divides documents into chunks, A database device that searches for the chunk closest to the query, Equipped with, The aforementioned database device, A clustering unit divides the first vector data, which is the vectorized data of the aforementioned chunks, into multiple clusters by clustering, and determines the centroid of each cluster. A list creation unit creates a list of first vector data belonging to each cluster and first vector data included in a predetermined region that straddles the boundary of each cluster, A search unit compares the first vector data with the second vector data created for the query based on the aforementioned list, and when searching for the first vector data that is close to the second vector data, it compares using the first vector data included in the cluster to which the second vector data belongs, and the first vector data included in the predetermined area set with respect to the boundary of the cluster to which the second vector data belongs. A database system equipped with the following features.

Images