Vector indexing method, device and electronic device based on distributed database

The vector indexing method optimizes query efficiency in distributed databases by pre-selecting sub-areas based on centroid vectors and distribution information, reducing node interactions and ensuring accurate vector search.

CN118796837BActive Publication Date: 2024-12-06SUZHOU JIBEISI DATA TECHNOLOGY CO LTD +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411275743.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-12-06
Estimated Expiration
2044-09-12

AI Technical Summary

Technical Problem

The challenge of efficiently querying unstructured data in distributed databases is exacerbated by the large number of data nodes, leading to inefficient interactive operations.

Method used

A vector indexing method that determines a target storage area and multiple sub-areas to be queried using a sharding storage algorithm and centroid vectors, optimizing the number of vectors to be obtained from each sub-area based on distribution information, thereby reducing the number of interacting nodes and improving query efficiency.

Benefits of technology

This approach ensures accurate vector search while minimizing data node interactions, enhancing query speed and efficiency in distributed database systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118796837B_ABST
    Figure CN118796837B_ABST
Patent Text Reader

Abstract

The present invention provides a vector indexing method, device, and electronic device based on a distributed database, which can be applied to the field of database technology. The method includes: in response to receiving a data query request sent by a client, obtaining a query vector, wherein the query vector is converted from unstructured data; using a sharding storage algorithm of a distributed database, determining a storage area for storing the query vector, and using the storage area as a target area to be queried; determining multiple sub-areas to be queried that match the query vector according to a centroid vector in vector index information; determining the number of vectors to be obtained that match each sub-area to be queried according to first distribution information between the target area to be queried and the multiple sub-areas to be queried; and obtaining multiple target vectors from the multiple sub-areas to be queried according to the query vector based on the number of vectors obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of database technology, and more specifically, to a vector indexing method, device, and electronic device based on a distributed database. Background Art

[0002] With the rapid development of technologies such as the Internet, big data, and artificial intelligence, unstructured data has begun to grow explosively. How to quickly and accurately retrieve and query this data has brought increasingly severe challenges to the traditional database industry. Embedding technology, which is widely used in artificial intelligence algorithms, abstracts unstructured data into vector data. As a result, the retrieval of unstructured data is converted into a vector retrieval process, which brings a new method to solve this problem.

[0003] Currently, distributed database clusters store vectorized unstructured data on different data nodes to achieve large-scale data storage and high-concurrency queries. Vector indexing of unstructured data based on distributed databases generally sends a vector acquisition request to each data node to achieve unstructured data query.

[0004] However, due to the large number of data nodes involved in a distributed database cluster, interactive operations between multiple data nodes will affect query efficiency. Summary of the invention

[0005] In view of this, the present invention provides a vector indexing method, device, and electronic device based on a distributed database.

[0006] One aspect of the present invention provides a vector indexing method based on a distributed database, comprising: in response to receiving a data query request sent by a client, obtaining a query vector, wherein the query vector is converted from unstructured data; using a sharding storage algorithm of the distributed database, determining a storage area for storing the query vector, and using the storage area as a target area to be queried; determining multiple sub-areas to be queried that match the query vector according to a centroid vector in vector index information; determining the number of vectors to be obtained that match each sub-area to be queried according to first distribution information between the target area to be queried and the multiple sub-areas to be queried; and obtaining multiple target vectors from the multiple sub-areas to be queried according to the query vector based on the number of vectors obtained.

[0007] According to an embodiment of the present invention, multiple sub-regions to be queried that match the query vector are determined based on the centroid vector in the vector index information, including: calculating the similarity between the query vector and each centroid vector in the distributed database, wherein each centroid vector corresponds to a storage sub-region; and determining multiple sub-regions to be queried based on the similarity and based on the storage sub-regions to which the multiple centroid vectors respectively correspond.

[0008] According to an embodiment of the present invention, based on the first distribution information between the target area to be queried and the multiple sub-areas to be queried, determining the number of vectors to be obtained that match each sub-area to be queried includes: determining a first preset number that matches each sub-area to be queried; determining a quantity weight corresponding to each sub-area to be queried based on the first distribution information between the target area to be queried and the multiple sub-areas to be queried; and determining the number of vectors to be obtained that match each sub-area to be queried based on the quantity weight and the first preset number.

[0009] According to an embodiment of the present invention, determining the first preset number that matches each sub-region to be queried includes: determining the first preset number that matches each sub-region to be queried based on a predetermined rule.

[0010] According to an embodiment of the present invention, determining a first preset number corresponding to each sub-region to be queried includes: obtaining regional distance distribution information in the vector index information, wherein the regional distance distribution information includes the distance between the vector stored in each data node and the centroid vector; based on the regional distance distribution information, determining a preset number of first candidate data nodes from each sub-region to be queried to obtain a set of candidate data nodes; determining a preset number of second candidate data nodes that are most similar to the query vector from the set of candidate data nodes; and determining the first preset number that matches each sub-region to be queried based on second distribution information of the second candidate data nodes in multiple sub-regions to be queried.

[0011] According to an embodiment of the present invention, a first preset number matching each sub-region to be queried is determined based on second distribution information of second candidate data nodes in multiple sub-regions to be queried, including: determining a second preset number of second candidate data nodes belonging to each sub-region to be queried based on second distribution information of second candidate data nodes in multiple sub-regions to be queried; and determining a first preset number matching each sub-region to be queried based on a ratio of each second preset number to the preset number.

[0012] According to an embodiment of the present invention, based on the number of vector acquisitions, multiple target vectors are acquired from multiple sub-regions to be queried according to a query vector, including: for each sub-region to be queried, according to the similarity between the query vector and vectors stored in multiple data nodes in the sub-region to be queried, determining the number of target data nodes for vector acquisition; sending a vector acquisition request to each target data node, wherein each target data node stores a target vector; and receiving the target vector corresponding to the vector acquisition request from each target data node.

[0013] According to an embodiment of the present invention, the method further includes: sorting the multiple target vectors according to the similarities between the multiple target vectors and the query vector; and sending the sorted multiple target vectors to the client.

[0014] Another aspect of the present invention provides a vector indexing device based on a distributed database, characterized in that the device includes: an acquisition module, used to obtain a query vector in response to receiving a data query request sent by a client, wherein the query vector is obtained by converting unstructured data; a first determination module, used to determine a storage area for storing the query vector using a sharding storage algorithm of the distributed database, and use the storage area as a target area to be queried; a second determination module, used to determine a plurality of sub-areas to be queried that match the query vector according to a centroid vector in vector index information; a third determination module, used to determine the number of vector acquisitions that match each sub-area to be queried according to first distribution information between the target area to be queried and the plurality of sub-areas to be queried; and a query module, used to acquire a plurality of target vectors from the plurality of sub-areas to be queried according to the query vector based on the number of vector acquisitions.

[0015] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0016] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0017] Another aspect of the present invention provides a computer program product, the computer program product comprising computer executable instructions, and the instructions are used to implement the above method when being executed.

[0018] The embodiments of the present invention implement vector indexing of unstructured data from two perspectives: storage and query. On the one hand, from the query perspective, multiple sub-areas to be queried are pre-selected based on the centroid vector to avoid data interaction with all data nodes in the distributed database, thereby reducing the number of interacting nodes. At the same time, since the centroid vector characterizes the overall characteristics of the sub-area to be queried, the predetermined sub-area to be queried can also ensure vector accuracy. On the other hand, from the storage perspective, the influence of the sharding rules and architecture of the distributed database is taken into account in combination with the sharding storage algorithm of the distributed database. Thus, the target area to be queried determined from the storage perspective and the multiple sub-areas to be queried determined from the query perspective are combined to determine the number of vector acquisitions for each sub-area to be queried. While ensuring the accuracy of the vector search, a smaller number of data nodes can be more accurately determined from the sub-area to be queried, and requests can be sent to a smaller number of data nodes to improve query efficiency and query speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0020] Figure 1 An exemplary system architecture of a distributed database-based vector index to which the present invention can be applied is shown.

[0021] Figure 2 A flow chart of a vector indexing method based on a distributed database according to an embodiment of the present invention is shown.

[0022] Figure 3 A schematic diagram of a distributed database storage structure according to an embodiment of the present invention is shown.

[0023] Figure 4 A schematic diagram of vector index interaction based on a distributed database according to an embodiment of the present invention is shown.

[0024] Figure 5 A block diagram of a vector indexing device based on a distributed database according to an embodiment of the present invention is shown.

[0025] Figure 6 A block diagram of an electronic device suitable for a vector indexing method based on a distributed database according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0029] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0030] In order to at least partially reduce the number of interactions between data nodes within a database during vector indexing and improve query efficiency, the present invention provides a vector indexing method for a distributed database, comprising: in response to receiving a data query request sent by a client, obtaining a query vector, wherein the query vector is converted from unstructured data; using a sharding storage algorithm of a distributed database, determining a storage area for storing the query vector, and using the storage area as a target area to be queried; determining multiple sub-areas to be queried matching the query vector according to a centroid vector in vector index information; determining the number of vectors to be obtained matching each sub-area to be queried according to first distribution information between the target area to be queried and the multiple sub-areas to be queried; and obtaining multiple target vectors from the multiple sub-areas to be queried according to the query vector based on the number of vectors obtained.

[0031] Figure 1 An exemplary system architecture of a vector index based on a distributed database to which the present invention can be applied is shown. It should be noted that: Figure 1The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention may not be used in other devices, systems, environments or scenarios.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first server 101 , a second server 102 , a third server 103 and a database operating system 104 .

[0033] The first server 101, the second server 102, and the third server 103 may be coordination nodes for coordinating the internal task allocation of the distributed database, or may be data nodes for storing data in the distributed database. The database operating system 104 may include a server cluster for unified management of multiple databases and servers.

[0034] The network is used as a medium for providing communication links between the first server 101, the second server 102, the third server 103 and the database operating system 104. The network may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0035] It should be noted that the distributed database-based vector indexing method provided in the embodiment of the present invention can generally be executed by the database operating system 104. Accordingly, the distributed database-based vector indexing device provided in the embodiment of the present invention can generally be set in a hardware device that carries the database operating system 104.

[0036] Alternatively, the vector indexing method based on a distributed database provided in an embodiment of the present invention can also be executed by the coordination nodes in the first server 101, the second server 102, and the third server 103. Accordingly, the vector indexing device based on a distributed database can also be set in the coordination nodes in the first server 101, the second server 102, and the third server 103.

[0037] Figure 2 A flow chart of a vector indexing method based on a distributed database according to an embodiment of the present invention is shown.

[0038] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0039] In operation S210 , in response to receiving a data query request sent by a client, a query vector is obtained, wherein the query vector is obtained by converting unstructured data.

[0040] In operation S220, a shard storage algorithm of a distributed database is used to determine a storage area for storing the query vector, and the storage area is used as a target area to be queried.

[0041] In operation S230, a plurality of to-be-queried sub-regions matching the query vector are determined according to the centroid vector in the vector index information.

[0042] In operation S240 , the number of vectors matching each sub-region to be queried is determined according to first distribution information between the target region to be queried and the plurality of sub-regions to be queried.

[0043] In operation S250 , based on the number of vector acquisitions, a plurality of target vectors are acquired from a plurality of to-be-queried sub-regions according to the query vector.

[0044] According to an embodiment of the present invention, unstructured data refers to data that does not conform to the rules of a predetermined data format. For example, unstructured data may be a piece of data that is randomly mixed with text and numbers.

[0045] The user can send a data query request to the database operating system through the client, and the database operating system sends the data query request to the coordinator in the distributed database, that is, the coordinating node. Then, the coordinator obtains the query vector in response to the data query request sent by the client, and performs subsequent operations based on the obtained query vector.

[0046] In an embodiment of the present invention, the query vector is obtained by converting unstructured data. The query vector may be obtained after a database operating system performs a conversion operation, or may be obtained after a coordinator performs a conversion operation.

[0047] According to an embodiment of the present invention, a distributed database generally includes a plurality of data nodes for storing a plurality of data. Specifically, the partition structure of the distributed database from top to bottom is: storage area-storage sub-area-data node, the storage area includes a plurality of storage sub-areas, and the storage sub-area includes a plurality of data nodes. The distributed database used in the present invention may be a distributed database cluster.

[0048] In the case of pre-classified storage, multiple vectors in each storage area and multiple vectors in each storage sub-area are relatively similar. In general, each data node stores a vector. For each storage sub-area, a centroid vector can be used to characterize the overall characteristics of multiple vectors in the storage sub-area as a whole. The centroid vector can be stored in an actual data node as an actual vector, such as selecting the centroid vector from multiple vectors stored in each storage sub-area. The centroid vector can also be a virtual vector, such as obtained by averaging all vectors in the storage sub-area.

[0049] For each data to be stored, a data node for storing the data to be stored is determined by a predefined distribution rule, and then the data to be stored is sent to the data node. The predefined distribution rule is also a shard storage algorithm, for example, a consistent hashing algorithm.

[0050] The embodiment of the present invention implements the search of the query vector from two perspectives: storage and query. Specifically, for operation S220 in the embodiment of the present invention, in the process of searching for the vector most similar to the query vector, the query vector is used as the data to be stored, and the storage area for storing the query vector is preliminarily located from the storage perspective, and is used as the target area to be queried. For operation S230, based on the query perspective, multiple sub-areas to be queried that match the query vector are searched.

[0051] According to an embodiment of the present invention, the coordinator may store vector index information of each storage sub-region, and the vector index information includes the centroid vector of each storage sub-region. Alternatively, the vector index information may also include the centroid vector of each storage sub-region, the regional distance distribution information of multiple data nodes in each storage sub-region; or, the name of the data node where the centroid vector is located.

[0052] Therefore, in operation S230, based on the similarity between the centroid vector and the query vector, that is, the distance, multiple storage sub-regions matching the query vector can be determined from multiple storage sub-regions of the distributed database, and the storage sub-regions matching the query vector are used as sub-regions to be queried.

[0053] It should be noted that the multiple storage sub-areas to be searched may belong to the same storage area or different storage areas, that is, the target area to be queried may include all sub-areas to be queried or only some of the sub-areas to be queried.

[0054] Therefore, for operation S240, the number of vectors matched with each sub-region to be queried can be determined based on the first distribution information between the target region to be queried and the plurality of sub-regions to be queried. Specifically, for the sub-region to be queried belonging to the target region to be queried, the number of vectors obtained is relatively large; for the sub-region to be queried that does not belong to the target region to be queried, the number of vectors obtained is relatively small.

[0055] According to an embodiment of the present invention, after determining multiple sub-regions to be queried and the number of vector acquisitions corresponding to each sub-region to be queried, multiple target vectors may be acquired from the multiple sub-regions to be queried according to the query vector based on the number of vector acquisitions.

[0056] For example, the vectors stored in each data node in the sub-region to be queried are sorted according to the similarity with the query vector, and then the vectors are selected in descending order of similarity to obtain the number of target vectors. Alternatively, vectors may be randomly obtained from each sub-region to be queried to obtain the number of target vectors.

[0057] The embodiments of the present invention implement vector indexing of unstructured data from two perspectives: storage and query. On the one hand, from the query perspective, multiple sub-areas to be queried are pre-selected based on the centroid vector to avoid data interaction with all data nodes in the distributed database, thereby reducing the number of interacting nodes. At the same time, since the centroid vector characterizes the overall characteristics of the sub-area to be queried, the predetermined sub-area to be queried can also ensure vector accuracy. On the other hand, from the storage perspective, the influence of the sharding rules and architecture of the distributed database is taken into account in combination with the sharding storage algorithm of the distributed database. Thus, the target area to be queried determined from the storage perspective and the multiple sub-areas to be queried determined from the query perspective are combined to determine the number of vector acquisitions for each sub-area to be queried. While ensuring the accuracy of the vector search, a smaller number of data nodes can be more accurately determined from the sub-area to be queried, and requests can be sent to a smaller number of data nodes to improve query efficiency and query speed.

[0058] Figure 3 A schematic diagram of a distributed database storage structure according to an embodiment of the present invention is shown.

[0059] like Figure 3 As shown, storage area 301 includes multiple storage sub-areas, such as storage sub-area 302. Each storage sub-area includes a data node where the centroid vector is located and multiple data nodes for storing vectors. For example, storage sub-area 302 includes data node 303 where the centroid vector is located, and data node 304 for storing vectors. In addition, storage area 301 may include coordinator 305, or coordinator node 305 may be outside storage area 301.

[0060] According to an embodiment of the present invention, for operation S230, multiple sub-regions to be queried that match the query vector are determined based on the centroid vector in the vector index information, including: calculating the similarity between the query vector and each centroid vector in the distributed database, wherein each centroid vector corresponds to a storage sub-region; and determining multiple sub-regions to be queried based on the similarity and based on the storage sub-regions to which the multiple centroid vectors respectively correspond.

[0061] According to an embodiment of the present invention, a variety of distance measurement algorithms may be used to calculate the similarity between the query vector and the centroid vector, for example, cosine similarity, inner product, Euclidean distance, and the like.

[0062] After calculating the similarity between the query vector and each centroid vector, multiple centroid vectors most similar to the query vector are determined in descending order of similarity, and the storage sub-regions corresponding to the multiple centroid vectors most similar to the query vector are used as sub-regions to be queried.

[0063] The embodiment of the present invention uses pre-constructed vector index information and utilizes the similarity between the centroid vector in the vector index information and the query vector to perform preliminary screening of the sub-areas to be queried through the coordinator before interacting with the data nodes, which not only preliminarily reduces the number of data nodes for subsequent interactions, but also ensures the accuracy of the screened target vector.

[0064] According to an embodiment of the present invention, for operation S240, the number of vectors matching each sub-region to be queried is determined based on the first distribution information between the target region to be queried and the multiple sub-regions to be queried, including: determining a first preset number matching each sub-region to be queried; determining a quantity weight corresponding to each sub-region to be queried based on the first distribution information between the target region to be queried and the multiple sub-regions to be queried; and determining the number of vectors matching each sub-region to be queried based on the quantity weight and the first preset number.

[0065] According to an embodiment of the present invention, in the process of determining the number of vectors to be obtained, a first preset number matching each sub-region to be queried is preliminarily determined. The first preset numbers of the sub-regions to be queried may be the same or different.

[0066] The first distribution information between the target area to be queried and the multiple sub-areas to be queried may include the inclusion relationship between the multiple sub-areas to be queried and the target area to be queried. For example, the first distribution information may include distribution inside or distribution outside, respectively indicating that the sub-area to be queried belongs to the target area to be queried or does not belong to the target area to be queried.

[0067] The number of sub-regions to be queried that are within the distribution has a higher weight, while the number of sub-regions to be queried that are outside the distribution has a lower weight. This is because the algorithm used for similarity calculation is generally different from the shard storage algorithm of the distributed database. When both the storage and query perspectives determine that a sub-region to be queried is similar to the query vector, the multiple vectors stored in the sub-region to be queried may be more similar to the query vector. Therefore, more vectors can be selected to ensure the accuracy of the vector index.

[0068] According to an embodiment of the present invention, the quantity weight may be multiplied by the first preset number to obtain the vector acquisition number. If the calculated vector acquisition number is not an integer, it may be processed by rounding up or rounding down.

[0069] According to an embodiment of the present invention, determining the first preset number that matches each sub-region to be queried includes: determining the first preset number that matches each sub-region to be queried based on a predetermined rule.

[0070] According to an embodiment of the present invention, multiple sub-regions to be queried are determined based on the overall features of the sub-region to be queried, that is, only the centroid vector is used, and the first preset number in each sub-region to be queried may not be processed.

[0071] For example, the first preset number corresponding to each sub-region to be queried is the same, which is a preset number, such as k.

[0072] Alternatively, the distance between the centroid vector and the query vector may also be considered, and according to a predetermined rule, more first preset numbers may be determined for sub-regions to be queried that are close to the query vector. It is understood that the more similar the centroid vector is to the query vector, the larger the first preset number of query sub-regions corresponding to the centroid vector.

[0073] According to an embodiment of the present invention, determining a first preset number corresponding to each sub-region to be queried also includes: obtaining regional distance distribution information in the vector index information, wherein the regional distance distribution information includes the distance between the vector stored in each data node and the centroid vector; based on the regional distance distribution information, determining a preset number of first candidate data nodes from each sub-region to be queried to obtain a set of candidate data nodes; determining a preset number of second candidate data nodes that are most similar to the query vector from the set of candidate data nodes; and determining the first preset number that matches each sub-region to be queried based on the second distribution information of the second candidate data nodes in the multiple sub-regions to be queried.

[0074] According to the embodiment of the present invention, in addition to starting from the overall characteristics of the sub-region to be queried, multiple data nodes in the sub-region to be queried are further considered.

[0075] In an embodiment of the present invention, the regional distance distribution information includes the distance between the vector stored in each data node and the centroid vector, which can be calculated by a variety of distance measurement algorithms.

[0076] For each sub-region to be queried, a preset number of first candidate data nodes closest to the centroid vector are selected from the sub-region to be queried. The first candidate data nodes selected from each sub-region to be queried are summarized to obtain a candidate data node set. At this time, by calculating the similarity between each first candidate data node in the candidate data node set and the query vector, a preset number of second candidate data nodes are selected from the candidate data node set in descending order of similarity.

[0077] For example, after determining N sub-regions to be queried, k closest first candidate data nodes are selected from each sub-region to be queried, and the final candidate data node set includes k*N first candidate data nodes. After calculating the similarity between the k*N first candidate data nodes and the query vector, k second candidate data nodes that are most similar to the query vector can be selected from the k*N first candidate data nodes. It should be noted that the number of first candidate data nodes selected from each sub-region to be queried and the number of second candidate data nodes selected from the candidate data nodes can be the same or different.

[0078] According to an embodiment of the present invention, the second distribution information of the second candidate data node in the multiple sub-regions to be queried may include the sub-region to be queried to which the second candidate data node belongs. It can be understood that, through the second distribution information of the second candidate data node in the multiple sub-regions to be queried, the distance between the sub-region to be queried and the query vector can be further determined from the perspective of the data node lower than the centroid vector, so that when the number of data nodes to be selected is limited, the appropriate first preset number can be selected from each sub-region to be queried as much as possible according to the distance distribution.

[0079] In an embodiment of the present invention, the more the number of second candidate data nodes included in a sub-region to be queried is, the larger the first preset number of the sub-region to be queried is.

[0080] According to an embodiment of the present invention, a first preset number matching each sub-region to be queried is determined based on second distribution information of second candidate data nodes in multiple sub-regions to be queried, including: determining a second preset number of second candidate data nodes belonging to each sub-region to be queried based on second distribution information of second candidate data nodes in multiple sub-regions to be queried; and determining a first preset number matching each sub-region to be queried based on a ratio of each second preset number to the preset number.

[0081] In an embodiment of the present invention, a second preset number of second candidate data nodes belonging to each sub-region to be queried may be counted according to the second distribution information of the second candidate data nodes in the plurality of sub-regions to be queried. Then, according to the ratio of each second preset number to the preset number, the distance between the sub-region to be queried and the query vector may be more accurately characterized.

[0082] According to an embodiment of the present invention, based on the number of vector acquisitions, multiple target vectors are acquired from multiple sub-regions to be queried according to a query vector, including: for each sub-region to be queried, according to the similarity between the query vector and vectors stored in multiple data nodes in the sub-region to be queried, determining the number of target data nodes for vector acquisition; sending a vector acquisition request to each target data node, wherein each target data node stores a target vector; and receiving the target vector corresponding to the vector acquisition request from each target data node.

[0083] After determining the sub-region to be queried and the number of vector acquisitions in the sub-region to be queried, the vector acquisition number target data nodes can be determined in descending order of similarity according to the similarity between the vector stored in each data node in the sub-region to be queried and the query vector. Then, the coordinator obtains the vector acquisition number target vectors from the vector acquisition number target data nodes by interacting with the vector acquisition number target data nodes.

[0084] By adopting the embodiment of the present invention, after determining the target data node, the coordinator only interacts with the vectors in each query sub-area to obtain a number of target vectors to obtain the vectors stored therein, which greatly reduces the number of interactions with the data nodes and improves query efficiency.

[0085] Figure 4 A schematic diagram of vector index interaction based on a distributed database according to an embodiment of the present invention is shown.

[0086] like Figure 4 As shown, before performing a vector search for a query vector, that is, in the construction phase of a distributed database, vector index information is constructed and cached in the coordinator. Thus, when performing a vector search for a query vector, the coordinator can use the cached vector index information to control the search range of the vector.

[0087] like Figure 4 As shown, the coordinator obtains a query vector in response to receiving a data query request sent by a client. On the one hand, the storage area for storing the query vector is determined by using the shard storage algorithm of the distributed database; on the other hand, the centroid vector in the cached vector index information is used to determine multiple sub-areas to be queried that match the query vector. In the process of fusing the query and storing the information, the number of vectors to be obtained that match each sub-area to be queried is determined based on the first distribution information between the target area to be queried and the multiple sub-areas to be queried. Finally, based on the number of vectors obtained, the coordinator obtains multiple target vectors from the multiple sub-areas to be queried according to the query vector.

[0088] Then, after obtaining the multiple target vectors, the coordinator returns the multiple target vectors to the client in a preset order. For example, the preset order may be an order in which the vectors are sorted according to the similarity.

[0089] According to an embodiment of the present invention, during the construction phase of a distributed database, constructing vector index information may include the following three steps.

[0090] Step 1: Estimate the amount of data that may be stored in the distributed database in advance to trigger the creation of the vector index. The parameters of the vector index creation process include: the total number of tables nlists, the number of data nodes notes, the number of rows of the maximum vector data in each data node rows, and the number of storage sub-areas nprobes. nlists=notes*sqrt(rows), nprobes=sqrt(nlists), sqrt() is the square root function. After that, after the vector index is created, the coordinator forwards the index creation request and parameters to each data shard node. Generally, this creation operation is performed after there is a certain amount of data in the distributed database to avoid a large deviation between the selected centroid vector and the final centroid vector.

[0091] Step 2: Create a vector index. Creating a vector index includes initializing the centroid vector and calculating the distance distribution information. Calculating the distance distribution information is to calculate the distance between the centroid vector and each data node in the storage sub-area. In the process of initializing the centroid vector, Knuth's random sampling theory can be used to select a certain number of sampling points from the memory block recorded in the vector data table; then, the kmeans++ algorithm is used to cluster the random sampling points to select high-quality centroid vectors. In the process of calculating the distance distribution information, according to the distance between the vector recorded in the table in the distributed database and each centroid vector, it is optimized and updated to the corresponding storage sub-partition, and the distance between the vector and the centroid vector in the storage sub-partition is updated.

[0092] Step 3: The data node sends the selected vector index information back to the coordinator and caches it in the coordinator's global table.

[0093] Step 4: After creating the index, when you store new vectors in the distributed database, you can update the vector index information and the information in the global table synchronously. If the subsequent update volume is too large, you can rebuild the vector index information to optimize the centroid vector and improve the accuracy of the query.

[0094] According to an embodiment of the present invention, the method further includes: sorting the multiple target vectors according to the similarities between the multiple target vectors and the query vector; and sending the sorted multiple target vectors to the client.

[0095] By sorting multiple target vectors obtained from data nodes, the differences between the multiple target vectors can be intuitively fed back when the client displays them to the user.

[0096] The present invention provides vector expansion capabilities for distributed database clusters, bringing faster retrieval speed and more accurate similarity matching to the explosively growing unstructured data, which can help users better mine the value of data. At the same time, the present invention also provides a hybrid query of stored scalars and vectors, allowing users to perform more complex query requests, providing more accurate and diverse search results, and meeting different user query needs and preferences.

[0097] Figure 5 A block diagram of a vector indexing device based on a distributed database according to an embodiment of the present invention is shown.

[0098] like Figure 5 As shown, the vector indexing device 500 based on a distributed database includes an acquisition module 510 , a first determination module 520 , a second determination module 530 , a third determination module 540 and a query module 550 .

[0099] The acquisition module 510 is configured to acquire a query vector in response to receiving a data query request sent by a client, wherein the query vector is obtained by converting unstructured data.

[0100] The first determination module 520 is used to determine a storage area for storing the query vector by using a shard storage algorithm of a distributed database, and use the storage area as a target area to be queried.

[0101] The second determination module 530 is used to determine a plurality of to-be-queried sub-regions matching the query vector according to the centroid vector in the vector index information.

[0102] The third determination module 540 is used to determine the number of vectors matching each sub-region to be queried according to the first distribution information between the target region to be queried and the multiple sub-regions to be queried.

[0103] The query module 550 is used to obtain the number based on the vector and obtain multiple target vectors from multiple sub-regions to be queried according to the query vector.

[0104] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0105] For example, any of the acquisition module 510, the first determination module 520, the second determination module 530, the third determination module 540, and the query module 550 can be combined in one module / unit / sub-unit for implementation, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 510, the first determination module 520, the second determination module 530, the third determination module 540, and the query module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the acquisition module 510, the first determination module 520, the second determination module 530, the third determination module 540 and the query module 550 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0106] It should be noted that the device part in the embodiment of the present invention corresponds to the method part in the embodiment of the present invention. The description of the device part specifically refers to the method part, which will not be repeated here.

[0107] Figure 6 A block diagram of an electronic device suitable for a vector indexing method based on a distributed database according to an embodiment of the present invention is shown. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0108] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0109] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0110] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0111] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0112] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0113] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device.

[0114] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .

[0115] An embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0116] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present invention are executed. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0117] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0118] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0120] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A vector indexing method based on a distributed database, characterized in that: The method comprises: In response to receiving a data query request sent by a client, obtaining a query vector, wherein the query vector is obtained by converting unstructured data; Determine a storage area for storing the query vector by using a shard storage algorithm of a distributed database, and use the storage area as a target area to be queried; Determining, according to the centroid vector in the vector index information, a plurality of sub-regions to be queried that match the query vector; Determining the number of vectors matching each of the sub-regions to be queried according to first distribution information between the target region to be queried and the plurality of sub-regions to be queried; Based on the vector acquisition number, acquiring multiple target vectors from the multiple sub-areas to be queried according to the query vector; The step of determining, according to the centroid vector in the vector index information, a plurality of sub-regions to be queried that match the query vector comprises: Calculating the similarity between the query vector and each centroid vector in the distributed database, wherein each centroid vector corresponds to a storage sub-region; and According to the similarity, based on the storage sub-regions corresponding to each of the plurality of centroid vectors, determining the plurality of sub-regions to be queried; The step of determining the number of vectors matching each of the sub-regions to be queried according to the first distribution information between the target region to be queried and the plurality of sub-regions to be queried includes: Determining a first preset number that matches each of the sub-regions to be queried; Determining a quantity weight corresponding to each of the sub-regions to be queried according to first distribution information between the target region to be queried and the plurality of sub-regions to be queried; and The number of vectors matched with each of the sub-regions to be queried is determined according to the quantity weight and the first preset number.

2. The method according to claim 1, characterized in that The determining of a first preset number matching each of the sub-regions to be queried includes: Based on a predetermined rule, a first preset number matching each of the sub-regions to be queried is determined.

3. The method according to claim 1, characterized in that The determining of the first preset number corresponding to each of the sub-areas to be queried further includes: Acquire regional distance distribution information in the vector index information, wherein the regional distance distribution information includes the distance between the vector stored in each data node and the centroid vector; Based on the regional distance distribution information, determining a preset number of first candidate data nodes from each of the sub-regions to be queried to obtain a candidate data node set; Determining a preset number of second candidate data nodes that are most similar to the query vector from the candidate data node set; and A first preset number matching each of the sub-regions to be queried is determined according to the second distribution information of the second candidate data nodes in the multiple sub-regions to be queried.

4. The method according to claim 3, characterized in that The determining, according to the second distribution information of the second candidate data nodes in the plurality of sub-areas to be queried, a first preset number matching each of the sub-areas to be queried comprises: Determine a second preset number of second candidate data nodes belonging to each of the sub-areas to be queried according to second distribution information of the second candidate data nodes in the multiple sub-areas to be queried; and According to the ratio of each of the second preset numbers to the preset number, a first preset number matching each of the sub-regions to be queried is determined.

5. The method according to claim 1, characterized in that The acquiring number based on the vector, acquiring a plurality of target vectors from the plurality of sub-regions to be queried according to the query vector, comprises: For each of the sub-regions to be queried, determining the number of target data nodes to be obtained by the vector according to the similarity between the query vector and the vectors stored in the plurality of data nodes in the sub-region to be queried; Sending a vector acquisition request to each of the target data nodes, wherein each of the target data nodes stores one of the target vectors; and A target vector corresponding to the vector acquisition request is received from each of the target data nodes.

6. The method according to claim 1, characterized in that The method further comprises: sorting the target vectors according to the similarities between the target vectors and the query vector; and The sorted plurality of target vectors are sent to the client.

7. A vector indexing device based on a distributed database, characterized in that: The device comprises: An acquisition module, configured to acquire a query vector in response to receiving a data query request sent by a client, wherein the query vector is obtained by converting unstructured data; A first determination module, configured to determine a storage area for storing the query vector by using a shard storage algorithm of a distributed database, and use the storage area as a target area to be queried; A second determination module is used to determine a plurality of to-be-queried sub-regions matching the query vector according to the centroid vector in the vector index information; A third determination module, configured to determine the number of vectors matched with each of the sub-regions to be queried according to first distribution information between the target region to be queried and the plurality of sub-regions to be queried; and A query module, configured to obtain a number based on the vector, and obtain a plurality of target vectors from the plurality of sub-areas to be queried according to the query vector; The second determination module is further used to: calculate the similarity between the query vector and each centroid vector in the distributed database, wherein each centroid vector corresponds to a storage sub-region; and determine the multiple sub-regions to be queried based on the storage sub-regions corresponding to the multiple centroid vectors according to the similarity; The third determination module is further used to: determine a first preset number that matches each of the sub-regions to be queried; determine a quantity weight corresponding to each of the sub-regions to be queried based on first distribution information between the target region to be queried and the multiple sub-regions to be queried; and determine the number of vectors to be obtained that match each of the sub-regions to be queried based on the quantity weight and the first preset number.

8. An electronic device, comprising: one or more processors; a memory for storing one or more programs, It is characterized in that when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Method and device for generating information

    CN110263209A

  • Distributed system, data aggregation method and computer readable storage medium

    CN113132267A

  • Multimedia content retrieval method and device, equipment and storage medium

    CN113495965A

  • Method and apparatus for searching multimedia content device, and storage medium

    US20210209146A1

  • Data query apparatus, method, and storage medium

    US20240086408A1