Database management methods and information processing systems

Abstract

This invention provides a database management method that reduces the amount of data in a vector database by suppressing the decrease in search accuracy. According to an embodiment, the database management method manages multiple clusters, each having a representative vector. In the database management method, when registering a first D-dimensional vector in the vector database, the cluster containing the representative vector closest to the first D-dimensional vector is determined, and D differences are calculated by subtracting the D elements contained in the representative vector of the determined cluster from the D elements contained in the first D-dimensional vector. In the database management method, the identifier of the determined cluster and the D differences are stored in the vector database as position information of the first D-dimensional vector.

Claims

1. A database management method for managing a vector database, the database management method comprising: Multiple clusters, each with a representative vector, are managed. The representative vector represents a representative point in a D-dimensional (D is an integer greater than 2) vector space and has D dimensions. When registering a first D-dimensional vector in the vector database, the cluster containing the representative vector closest to the first D-dimensional vector is determined as the cluster to which the first D-dimensional vector belongs. D differences are calculated, obtained by subtracting the D eigenvalues ​​contained in the representative vector of the determined cluster from the D eigenvalues ​​contained in the first D-dimensional vector. The identifier of the determined cluster and the D differences are stored in the vector database as positional information representing the position of the first D-dimensional vector in the D-dimensional vector space.

2. The database management method as described in claim 1, wherein each of the D feature values ​​contained in the first D-dimensional vector is represented by first data having a first bit number, and each of the D difference values ​​is represented by second data having a second bit number less than the first bit number.

3. The database management method as described in claim 2, wherein the first data is a bit string for a floating-point number containing a 1-bit code, an exponent having a third bit, and a mantissa having a fourth bit, and the second data is (1) a bit string for a floating-point number containing a 1-bit code, an exponent having a fifth bit less than the third bit, and a mantissa having a sixth bit less than the fourth bit, or (2) a bit string for a fixed-point number containing a 1-bit code, an integer having a seventh bit, and a fractional having an eighth bit, but not containing the exponent, wherein the sum of the seventh bit and the eighth bit is less than or equal to the fourth bit.

4. The database management method as described in claim 1 further includes: Create a new cluster with representative vectors; Determine a second D-dimensional vector from the set of D-dimensional vectors in a first cluster that is adjacent to the representative vector of the new cluster, whose distance to the representative vector of the new cluster is shorter than the distance to the representative vector of the first cluster; and perform a movement process to move the second D-dimensional vector from the first cluster to the new cluster, the movement process comprising: obtaining D first differences from the vector database, the D first differences being obtained by subtracting the D eigenvalues ​​contained in the representative vector of the first cluster from the D eigenvalues ​​contained in the second D-dimensional vector; and calculating D second differences, the D second differences being obtained by subtracting the D eigenvalues ​​contained in the representative vector of the new cluster from the D eigenvalues ​​contained in the representative vector of the first cluster; The process of calculating D third differences, which are obtained by adding the D second differences to the D first differences respectively; and the process of storing the identifier of the new cluster and the D third differences as information representing the position of the second D-dimensional vector in the D-dimensional vector space in the vector database.

5. The database management method as described in claim 2 further includes: When a new D-dimensional vector is added to the vector database where the distance to all representative vectors of the plurality of clusters exceeds the maximum value of the difference that can be expressed by the second data, a new cluster with representative vectors close to the new D-dimensional vector is created; D fourth differences are calculated, which are obtained by subtracting the D eigenvalues ​​contained in the representative vectors of the new cluster from the D eigenvalues ​​contained in the new D-dimensional vector; and the identifier of the new cluster and the D fourth differences are stored in the vector database as information representing the position of the new D-dimensional vector in the D-dimensional vector space.

6. The database management method as claimed in claim 1, comprising: receiving a query vector having D dimensions; when calculating the distance between the query vector and a third D-dimensional vector, which is one of the D-dimensional vectors registered in the vector database, obtaining D fifth differences from the vector database, the D fifth differences being obtained by subtracting the D feature values ​​contained in a representative vector of a second cluster belonging to the third D-dimensional vector from the D feature values ​​contained in the third D-dimensional vector; calculating D sixth differences, the D sixth differences being obtained by subtracting the D feature values ​​contained in the representative vector of the second cluster from the D feature values ​​contained in the query vector; and calculating the sum of squares of D seventh differences between the D fifth differences and the D sixth differences, or the square root of the sum of squares.

7. A database management method for managing a vector database, the database management method comprising: For each of the R sub-vector spaces (R is an integer greater than or equal to 2 but less than D) obtained by dividing a D-dimensional (D is an integer greater than or equal to 2) vector space, each having a dimension of D / R, multiple clusters with representative vectors are managed. Each representative vector represents a representative point within the sub-vector space and has dimensions of D / R. When registering a first D-dimensional vector in the vector database, the first D-dimensional vector is divided into R sub-vectors, each having dimensions of D / R. For each of the R sub-vectors, the cluster containing the representative vector of the sub-vector space corresponding to the sub-vector is determined as the cluster to which the sub-vector belongs. For each of the R sub-vectors, D / R differences are calculated, obtained by subtracting the D / R eigenvalues ​​contained in the representative vector of the determined cluster from the D / R eigenvalues ​​contained in the sub-vector. For each of the R sub-vectors, the identifier of the determined cluster and the D / R differences are stored in the vector database as position information representing the position of the sub-vector in the sub-vector space corresponding to the sub-vector.

8. The database management method as described in claim 7, wherein the D / R feature values ​​contained in each of the R subvectors are represented by first data having a first bit number, and each of the D / R differences is represented by second data having a second bit number less than the first bit number.

9. The database management method as described in claim 8, wherein the first data is a bit string for a floating-point number containing a 1-bit code, an exponent having a third bit, and a mantissa having a fourth bit, and the second data is (1) a bit string for a floating-point number containing a 1-bit code, an exponent having a fifth bit less than the third bit, and a mantissa having a sixth bit less than the fourth bit, or (2) a bit string for a fixed-point number containing a 1-bit code, an integer having a seventh bit, and a fractional having an eighth bit, but not containing the exponent, wherein the sum of the seventh bit and the eighth bit is less than or equal to the fourth bit.

10. The database management method as described in claim 7 further includes: Create new clusters for each of the R sub-vector spaces; For each of the R sub-vector spaces, from the set of sub-vectors belonging to the first cluster which has a representative vector close to the representative vector of the new cluster, determine a first sub-vector whose distance to the representative vector of the new cluster is shorter than its distance to the representative vector of the first cluster; and for each of the R sub-vector spaces, perform a movement process to move the first sub-vector from the first cluster to the new cluster, the movement process comprising: for each of the R sub-vector spaces, obtaining D / R first differences from the vector database, the D / R first differences being obtained by subtracting the D / R eigenvalues ​​contained in the representative vector of the first cluster from the D / R eigenvalues ​​contained in the first sub-vector; For each of the R sub-vector spaces, the process of calculating D / R second differences is performed, wherein the D / R second differences are obtained by subtracting the D / R feature values ​​contained in the representative vector of the new cluster from the D / R feature values ​​contained in the representative vector of the first cluster; for each of the R sub-vector spaces, the process of calculating D / R third differences is performed, wherein the D / R third differences are obtained by adding the D / R second differences to the D / R first differences; and for each of the R sub-vector spaces, the process of storing the identifier of the new cluster and the D / R third differences as position information representing the position of the first sub-vector within the sub-vector space in the vector database is performed.

11. The database management method as described in claim 8 further includes: When a new D-dimensional vector containing subvectors, i.e., the distance to all representative vectors of the plurality of clusters in at least one of the R subvector spaces exceeds the maximum value of the difference that can be expressed by the second data, is added to the vector database, for each of the R subvectors contained in the new D-dimensional vector, a new cluster with a representative vector close to the subvector is created in the corresponding subvector space; For each of the R sub-vectors contained in the new D-dimensional vector, D / R fourth differences are calculated, which are obtained by subtracting the D / R eigenvalues ​​contained in the representative vector of the new cluster in the corresponding sub-vector space from the D / R eigenvalues ​​contained in the sub-vectors; and for each of the R sub-vectors contained in the new D-dimensional vector, the identifier of the new cluster created in the corresponding sub-vector space and the D / R fourth differences are stored in the vector database as information representing the position of the sub-vector in the corresponding sub-vector space.

12. The database management method as described in claim 7 further includes: Receives a query vector with D dimensions; When calculating the distance between the query vector and the second D-dimensional vector, which is one of the D-dimensional vectors registered in the vector database, for each of the R sub-vector spaces, D / R fifth differences are obtained from the vector database. These D / R fifth differences are obtained by subtracting the D / R eigenvalues ​​contained in the representative vector of the cluster of sub-vectors belonging to the second D-dimensional vector corresponding to the sub-vector space from the D / R eigenvalues ​​contained in the sub-vectors of the second D-dimensional vector corresponding to the sub-vector space. The query vector is then divided into R sub-vectors, each with D / R dimensions. For each of the R sub-vector spaces, D / R sixth differences are calculated. These D / R sixth differences are obtained by subtracting the D / R eigenvalues ​​contained in the representative vector of the cluster of sub-vectors belonging to the second D-dimensional vector corresponding to the sub-vector space from the D / R eigenvalues ​​contained in the sub-vectors of the query vector corresponding to the sub-vector space. For each of the R sub-vector spaces, calculate the sum of squares of the D / R fifth differences corresponding to the sub-vector space and the D / R sixth differences corresponding to the sub-vector space; and calculate the total of the R sums of squares corresponding to each of the R sub-vector spaces.

13. An information processing system for managing a vector database, the information processing system comprising a processor, the processor being configured to manage multiple clusters, each having a representative vector, the representative vector representing a representative point in a D-dimensional (D is an integer greater than 2) vector space and having D dimensions; when registering a first D-dimensional vector in the vector database, determining the cluster of the multiple clusters having the representative vector closest to the first D-dimensional vector as the cluster to which the first D-dimensional vector should belong; calculating D differences, the D differences being obtained by subtracting the D eigenvalues ​​contained in the representative vector of the determined cluster from the D eigenvalues ​​contained in the first D-dimensional vector; and storing the identifier of the determined cluster and the D differences as position information representing the position of the first D-dimensional vector in the D-dimensional vector space in the vector database.

14. The information processing system of claim 13, wherein each of the D feature values ​​contained in the first D-dimensional vector is represented by first data having a first bit number, and each of the D difference values ​​is represented by second data having a second bit number less than the first bit number.

15. The information processing system of claim 14, wherein the first data is a bit string for a floating-point number containing a 1-bit code, an exponent having a third bit, and a mantissa having a fourth bit, and the second data is (1) a bit string for a floating-point number containing a 1-bit code, an exponent having a fifth bit less than the third bit, and a mantissa having a sixth bit less than the fourth bit, or (2) a bit string for a fixed-point number containing a 1-bit code, an integer having a seventh bit, and a fractional having an eighth bit, but not containing the exponent, wherein the sum of the seventh bit and the eighth bit is less than or equal to the fourth bit.

16. The information processing system of claim 13, wherein the processor is further configured to: create a new cluster having a representative vector; determine a second D-dimensional vector from a set of D-dimensional vectors attached to a first cluster that is close to the representative vector of the new cluster, wherein the distance to the representative vector of the new cluster is shorter than the distance to the representative vector of the first cluster; perform a movement process of moving the second D-dimensional vector from the first cluster to the new cluster, the movement process comprising: obtaining D first differences from the vector database, the D first differences being obtained by subtracting the D eigenvalues ​​contained in the representative vector of the first cluster from the D eigenvalues ​​contained in the second D-dimensional vector; and calculating D second differences, the D second differences being obtained by subtracting the D eigenvalues ​​contained in the representative vector of the new cluster from the D eigenvalues ​​contained in the representative vector of the first cluster. The process of calculating D third differences, which are obtained by adding the D second differences to the D first differences respectively; and the process of storing the identifier of the new cluster and the D third differences as information representing the position of the second D-dimensional vector in the D-dimensional vector space in the vector database.

17. The information processing system of claim 14, wherein the processor is further configured to, when appending a new D-dimensional vector to the vector database whose distance to all representative vectors of the plurality of clusters exceeds the maximum value of the difference that can be expressed by the second data, create a new cluster having representative vectors close to the new D-dimensional vector, calculate D fourth differences, the D fourth differences being obtained by subtracting the D eigenvalues ​​contained in the representative vectors of the new cluster from the D eigenvalues ​​contained in the new D-dimensional vector, and store the identifier of the new cluster and the D fourth differences as information representing the position of the new vector in the D-dimensional vector space in the vector database.

18. The information processing system of claim 13, wherein the processor is further configured to: receive a query vector having D dimensions; when calculating the distance between the query vector and a third D-dimensional vector, which is one of the D-dimensional vectors registered in the vector database, obtain D fifth differences from the vector database, the D fifth differences being obtained by subtracting the D feature values ​​contained in the representative vector of the second cluster, which is associated with the third D-dimensional vector, from the D feature values ​​contained in the third D-dimensional vector; calculate D sixth differences, the D sixth differences being obtained by subtracting the D feature values ​​contained in the representative vector of the second cluster, from the D feature values ​​contained in the query vector; and calculate the sum of squares of D seventh differences, or the square root of the sum of squares, between the D fifth differences and the D sixth differences.

Images