Database system and vector mixed retrieval method

Abstract

The one or more embodiments of the present application provide a database system and a vector mixed retrieval method. Indexes in the database include attribute sub-indexes for indicating vector sub-indexes, which are established based on attribute information subsets contained in data sets in the database, and vector sub-indexes for indicating data sets, which are established based on vector data contained in the data sets in each data set group. The attribute information subsets contained in the data sets in each data set group match. The method includes: obtaining a query request including target vector data and a target attribute information subset; in response to the query request, searching in the attribute sub-indexes based on the target attribute information subset to determine a target vector sub-index associated with the target attribute information subset; searching in the target vector sub-index based on the target vector data to determine a target data set containing the target vector data, and determining a query result corresponding to the query request based on the target data set.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present application relate to the technical field of database, and in particular, to a database system and a vector mixed retrieval method. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the presentation form of data is no longer limited to traditional structured data, and more and more data is presented in the form of pictures, videos, audio and other unstructured forms. Structured data is usually stored in the form of a two-dimensional table in a relational database. In order to effectively process and analyze unstructured data, the Embedding method is usually used to convert unstructured data into vector form data, that is, to perform vector representation on unstructured data, so as to facilitate subsequent similarity calculation and retrieval operations. Among them, Embedding refers to the process of mapping high-dimensional sparse data to low-dimensional dense vector space, which can effectively express the internal structural characteristics of the data. Vector similarity retrieval refers to calculating the distance (such as Euclidean distance, cosine similarity, etc.) between vectors. The smaller the distance between vectors, the higher the similarity between vectors.

[0003] In actual application scenarios, single vector similarity retrieval can only achieve similarity retrieval of unstructured data, while ignoring the structured attribute information attached to the unstructured data. Taking an e-commerce recommendation system as an example, vector similarity retrieval can only retrieve the goods that are most similar to the current goods picture in vision, but ignores the brand, price, category and other attribute information of the goods in the picture. Due to the lack of effective use of these structured attribute information, single vector similarity retrieval is usually difficult to meet the diversified retrieval needs in complex service scenarios. Therefore, it is expected to combine structured attribute information for filtering while performing vector similarity retrieval. SUMMARY

[0004] One or more embodiments of the present application provide technical solutions as follows:

[0005] The present application provides a database system, which comprises a database; wherein:

[0006] Each data set in the database contains vector data and an attribute information subset;

[0007] The index in the database includes an attribute sub-index established based on the attribute information subset, and a vector sub-index established based on the vector data contained in the data sets in each data set group; the attribute information subsets contained in the data sets in each data set group match;

[0008] The attribute sub-index is used to indicate the vector sub-index associated with each subset of attribute information; the vector sub-index is used to indicate the data set containing each vector data.

[0009] This application also provides a vector hybrid retrieval method; each data set in the database contains vector data and a subset of attribute information; the index in the database includes an attribute sub-index built based on the subset of attribute information, and a vector sub-index built based on the vector data contained in the data sets in each data set group; the subsets of attribute information contained in the data sets in each data set group are matched; the attribute sub-index is used to indicate the vector sub-index associated with each subset of attribute information; the vector sub-index is used to indicate the data set containing each vector data;

[0010] The method includes:

[0011] Obtain a query request for the database; wherein the query request includes target vector data and a subset of target attribute information;

[0012] In response to the query request, based on the subset of target attribute information, a search is performed in the attribute sub-index to determine the target vector sub-index associated with the subset of target attribute information;

[0013] Based on the target vector data, a search is performed in the target vector sub-index to determine the target data set containing the target vector data, and based on the target data set, the query result corresponding to the query request is determined.

[0014] This application also provides a vector hybrid retrieval device; each data set in the database contains vector data and a subset of attribute information; the index in the database includes an attribute sub-index built based on the subset of attribute information, and a vector sub-index built based on the vector data contained in the data sets in each data set group; the subsets of attribute information contained in the data sets in each data set group are matched; the attribute sub-index is used to indicate the vector sub-index associated with each subset of attribute information; the vector sub-index is used to indicate the data set containing each vector data;

[0015] The device includes:

[0016] The query acquisition module acquires a query request for the database; wherein the query request includes target vector data and a subset of target attribute information;

[0017] The attribute lookup module, in response to the query request, searches within the attribute sub-index based on the target attribute information subset to determine the target vector sub-index associated with the target attribute information subset;

[0018] The vector lookup module searches within the target vector sub-index based on the target vector data to determine the target data set containing the target vector data, and then determines the query result corresponding to the query request based on the target data set.

[0019] This application also provides an electronic device, including:

[0020] processor;

[0021] Memory used to store processor-executable instructions;

[0022] The processor executes the executable instructions to implement the steps of the method as described in any of the preceding descriptions.

[0023] This application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as described in any of the preceding claims.

[0024] In the above technical solution, each data set in the database can contain vector data and subsets of attribute information. The index can include an attribute sub-index built based on the subsets of attribute information contained in the data sets in the database, which indicates the vector sub-index associated with each subset of attribute information, and a vector sub-index built based on the vector data contained in the data sets in each data set group, which indicates the data set containing each vector data. The subsets of attribute information contained in the data sets in each data set group match. Accordingly, when a query request for the database is obtained, the target vector sub-index can be searched first based on the target subset of attribute information in the query request to determine the target vector sub-index associated with the target subset of attribute information. Then, the target vector data in the query request can be searched in the target vector sub-index to determine the target data set containing the target vector data. Finally, the query result corresponding to the query request can be determined based on the target data set.

[0025] Using the above method, for a dataset containing vector data and attribute information subsets stored in the database, data retrieval can be performed by combining vector indexes built based on vector data and attribute indexes built based on attribute information, thereby achieving hybrid vector retrieval. Attached Figure Description

[0026] The accompanying drawings used in the description of the exemplary embodiments will now be explained, wherein:

[0027] Figure 1 This is a schematic diagram of the architecture of a database system shown in an exemplary embodiment of this application.

[0028] Figure 2This is a schematic diagram illustrating the structure of a database index according to an exemplary embodiment of this application.

[0029] Figure 3 This is a schematic diagram illustrating the structure of an attribute sub-index according to an exemplary embodiment of this application.

[0030] Figure 4 This is a schematic diagram illustrating the structure of another attribute sub-index as shown in an exemplary embodiment of this application.

[0031] Figure 5 This is a schematic diagram illustrating the structure of another attribute sub-index as shown in an exemplary embodiment of this application.

[0032] Figure 6 This is a flowchart illustrating a vector hybrid retrieval method in an exemplary embodiment of this application.

[0033] Figure 7 This is a schematic diagram of the structure of a device shown in an exemplary embodiment of this application.

[0034] Figure 8 This is a block diagram illustrating a vector hybrid retrieval device according to an exemplary embodiment of this application. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this application. Rather, they are merely examples consistent with some aspects of one or more embodiments of this application.

[0036] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this application in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this application. Furthermore, a single step described in this application may be broken down into multiple steps in other embodiments; and multiple steps described in this application may be combined into a single step in other embodiments.

[0037] By combining vector similarity retrieval with filtering based on structured attribute information, this vector retrieval method can be called hybrid vector retrieval. In other words, hybrid vector retrieval refers to simultaneously retrieving unstructured data (the vector data of the original data) and structured data (the attribute information of the original data).

[0038] Vector hybrid retrieval can not only improve the relevance and accuracy of retrieval results, but also enhance the flexibility and practicality of the system, meeting the practical needs of multi-dimensional data fusion retrieval.

[0039] This application provides one or more embodiments of a technical solution for implementing vector hybrid retrieval. In this technical solution, each data set in the database may contain vector data and a subset of attribute information. The index may include an attribute sub-index built based on the subset of attribute information contained in the data set in the database, used to indicate the vector sub-index associated with each subset of attribute information, and a vector sub-index built based on the vector data contained in the data set in each data set group, used to indicate the data set containing each vector data. The subsets of attribute information contained in the data set in each data set group are matched. Accordingly, when a query request for the database is obtained, the target vector sub-index can be searched first based on the target subset of attribute information in the query request to determine the target vector sub-index associated with the target subset of attribute information. Then, the target vector data in the query request can be searched in the target vector sub-index to determine the target data set containing the target vector data. Finally, the query result corresponding to the query request can be determined based on the target data set.

[0040] Using the above method, for a dataset containing vector data and attribute information subsets stored in the database, data retrieval can be performed by combining vector indexes built based on vector data and attribute indexes built based on attribute information, thereby achieving hybrid vector retrieval.

[0041] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the architecture of a database system shown in an exemplary embodiment of this application.

[0042] A database system is a software system used to store, retrieve, and manage data. For example... Figure 1 As shown, a database system can include at least a database, a database management system, and database applications.

[0043] A database is a collection of data that is actually stored. The data in a database is organized according to a specific schema or model to facilitate retrieval and management.

[0044] A Database Management System (DBMS) is a software system that is the core component for controlling and managing databases. It is used to define, create, maintain, and operate databases. A DBMS provides a series of tools and interfaces for database management, such as interfaces for creating, reading, updating, and deleting data. The DBMS is primarily used to control the storage, retrieval, updating, and management of data, and is also responsible for data accuracy, consistency, security, and backup and recovery.

[0045] A database application refers to a front-end application that interacts with a database to implement specific service logic and functions. It can provide a user-facing interface, which may be a graphical user interface, command line, or other form of interface.

[0046] In practical applications, users can initiate specific operations on the database through database applications (such as index creation, data retrieval, and data query operations), and the database management system can then perform corresponding operations on the database.

[0047] In the context of databases, querying typically refers to executing specific database commands to retrieve data, searching typically refers to attempting to locate specific data, while retrieval encompasses a broader process of acquiring and potentially processing data.

[0048] The aforementioned database and database management system can be deployed on a server. This server can be a single physical host, a server cluster consisting of multiple independent physical hosts, or a virtual server, cloud server, etc., hosted by a host cluster. The aforementioned database application can be deployed on terminal devices that have established a network connection (including various types of wired or wireless networks) with the server. These terminal devices can be personal computers (PCs), desktop computers, laptops, tablets, smartphones, personal digital assistants (PDAs), etc.

[0049] In the database system illustrated in one or more embodiments of this application, the data stored in the database can be in the form of a data set. Specifically, a data set may contain a vector data and an attribute information set (referred to as an attribute information subset to distinguish it from the data set), and the attribute information subset may contain one or more attribute information corresponding to the vector data.

[0050] For example, in an e-commerce scenario, the data stored in the aforementioned database could be product-related data. For a single product, its related data can be processed into a dataset. This dataset may contain vector data corresponding to the product's image (e.g., the embedding vector corresponding to the image), as well as a subset of attribute information consisting of the product's attribute information (e.g., color information, price information, brand information, etc.).

[0051] For example, in a music playback scenario, the data stored in the aforementioned database could be song-related data. For a single song, the related data can be processed into a dataset. This dataset can contain vector data corresponding to the song's audio (e.g., the embedding vector corresponding to the image), as well as a subset of attribute information consisting of the song's attribute information (e.g., artist information, album information, number of likes, number of comments, etc.).

[0052] In database systems, indexes are introduced to speed up data retrieval. Indexes are typically created, managed, and used by the database management system, but are stored within the database itself. An index is a data structure, similar to a book's index, that helps the database quickly locate the required data without performing a full scan. Using indexes can significantly reduce the amount of data scanned for a query, speeding up the query process, especially when dealing with large amounts of data. However, indexes require additional storage space, and maintaining them also requires additional time when performing insert, update, or delete operations. Therefore, when designing a database, it is necessary to weigh the performance improvements brought by indexes against their maintenance costs.

[0053] For data in the form of data sets stored in the database mentioned above, since a data set can contain a vector data and a subset of attribute information, a vector index can be built based on the vector data, and an attribute index can be built based on the attribute information. The vector index and the attribute index can be used in combination to retrieve data, thereby realizing vector hybrid retrieval.

[0054] Please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the structure of a database index according to an exemplary embodiment of this application.

[0055] In this embodiment, as described above, the data stored in the database can be in the form of a data set. Specifically, a data set may contain a vector data and a subset of attribute information, wherein the subset of attribute information may contain one or more attribute information corresponding to the vector data.

[0056] Correspondingly, an index can be built based on the vector data contained in the dataset of the aforementioned database, serving as a vector index, and an index can be built based on a subset of attribute information contained in the dataset of the same database, serving as an attribute index. In practical applications, such as... Figure 2 As shown, the established vector index can be called a vector sub-index, and the established attribute index can be called an attribute sub-index. The vector sub-index and the attribute sub-index together form an index for realizing vector hybrid retrieval, which is stored in the database.

[0057] It should be noted that when building the aforementioned vector sub-index based on the vector data contained in the dataset of the database, the dataset in the database can first be grouped according to the subsets of attribute information it contains. This ensures that for any dataset group obtained through grouping, the subsets of attribute information contained in the datasets within that dataset group match. Specifically, matching the subsets of attribute information contained in two datasets can mean that corresponding attribute information within these subsets matches (e.g., two identical attribute information, or the similarity between two attribute information is greater than a specific threshold). For example, suppose the database stores three data sets (represented by "{}"): data set 1 = {vector=vector1, attributes={color="red", price=100, brand="brandX"}}, data set 2 = {vector=vector2, attributes={color="red", price=100, brand="brandX"}}, and data set 3 = {vector=vector3, attributes={color="black", price=200, brand="brandX"}}. Since the attribute information in the subset of attribute information in data set 1 is consistent with that in data set 2, but differs significantly from that in data set 3, data set 1 and data set 2 can be grouped into one data set group, and data set 3 into another. Subsequently, vector sub-indexes can be created based on the vector data contained in each data set group; that is, a vector sub-index can be created for each data set group.

[0058] To ensure the combined use of the aforementioned vector sub-indexes and attribute sub-indexes during data retrieval for hybrid vector retrieval, the attribute sub-indexes can be used to indicate vector sub-indexes associated with subsets of attribute information contained in each data set within the database. That is, assuming attribute information subset i is a subset of attribute information contained in data set i within the dataset database, using the attribute sub-index can quickly locate the vector sub-index associated with attribute information subset i (i.e., the vector sub-index built based on the vector data contained in the data set within the data set group to which data set i belongs). The aforementioned vector sub-indexes can also be used to indicate data sets containing various vector data; that is, assuming vector data j is vector data contained in data set j within the dataset database, using the vector sub-index associated with the subset of attribute information contained in data set j can quickly locate the data set containing vector data j (i.e., data set j).

[0059] When performing data retrieval, one can first use the aforementioned attribute sub-index to find the vector sub-index associated with the subset of attribute information in the query request. Then, based on the vector data in the query request, the vector sub-index can be used to find the data set containing that vector data. This achieves hybrid vector retrieval, simultaneously retrieving both vector data and attribute information.

[0060] In some embodiments, in order to optimize the storage space occupied by the above-mentioned attribute sub-index and speed up the retrieval of attribute information, thereby reducing the maintenance workload of the index composed of attribute sub-index and vector sub-index and accelerating the final data retrieval speed, the structure and idea of ​​Trie tree (also known as prefix tree or dictionary tree) can be used to build the attribute sub-index.

[0061] The original Trie tree is a tree-like data structure used for efficiently storing and retrieving collections of strings. Its basic structure is that a node represents a character, and the nodes along the path from the root node to any other node form a string. Its core idea is that strings sharing a common prefix share the same path. Therefore, the original Trie tree has the following characteristics: the root node does not contain a character; it is the entry point of the tree; a child node under the root node represents a character; the path from the root node to any other node forms a string; the node representing the ending character of a string can be a leaf node, which is a node without child nodes.

[0062] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the structure of an attribute sub-index according to an exemplary embodiment of this application.

[0063] like Figure 3 As shown, the aforementioned attribute sub-index can be an index based on a Trie tree. In the Trie tree that serves as the attribute sub-index, a node can represent one attribute information; the root node does not contain attribute information, but is the query entry point. Therefore, in this Trie tree, the nodes on the path from the root node to a certain node can form a set of attribute information, which can be a subset of the attribute information contained in a certain dataset. For example, suppose the path from the root node to the node representing attribute information N2 is: root node → node representing attribute information 12 → node representing attribute information 24 → node representing attribute information 33 → node representing attribute information N2, then the subset of attribute information formed by the nodes on this path is {attribute information 12, attribute information 24, attribute information 33, attribute information N2}.

[0064] Furthermore, in the Trie tree, which serves as the sub-index of the aforementioned attributes, a leaf node can represent the last attribute information in a subset of attribute information. Therefore, for any leaf node in the Trie tree, the leaf node can be associated with a vector sub-index of the attribute information subset formed by the nodes on the path from the root node to the leaf node in the Trie tree; that is, the correspondence between the leaf node and the vector sub-index can also be stored in the database, so that when the nodes forming the subset of attribute information in the query request are found in the Trie tree, the vector sub-index corresponding to the leaf node on the path formed by these nodes can be determined as the vector sub-index associated with the subset of attribute information.

[0065] In some embodiments, to further optimize the storage space occupied by the aforementioned attribute sub-index, when building the attribute sub-index based on a Trie tree, nodes representing attribute information of the same attribute type can be placed at the same level in the Trie tree. Here, attribute type refers to the attribute category, and attribute information refers to the specific content of the attribute. Taking an e-commerce scenario as an example, attribute type can be color, price, brand, etc., and attribute information can be red (attribute type is color), 100 (attribute type is price), and brand X (attribute type is brand). In the Trie tree serving as the attribute sub-index, for any two adjacent levels, the number of attribute information corresponding to the upper level is no greater than the number of attribute information corresponding to the lower level. The number of attribute information corresponding to a level can be the number of non-matching attribute information represented by the nodes at that level (i.e., the nodes located at that level). Taking an e-commerce scenario as an example, assuming there are 3 nodes at this level, and the attribute information represented by these 3 nodes is "red", "red", and "black", then the non-matching attribute information represented by these nodes is "red" and "black", with a quantity of 2.

[0066] Please refer to Figure 4 and Figure 5 , Figure 4 and Figure 5 These are schematic diagrams illustrating another attribute sub-index structure as shown in an exemplary embodiment of this application.

[0067] like Figure 4As shown, in the actual storage structure of the Trie tree, which serves as the sub-index of the aforementioned attributes, excluding the root node, the nodes at the first level represent attribute information of attribute type A, the nodes at the second level represent attribute type B, and the nodes at the third level represent attribute type C. Specifically, the number of mutually exclusive attribute information (i.e., attribute information A1) in attribute information of attribute type A is 1, the number of mutually exclusive attribute information (i.e., attribute information B1, attribute information B2, and attribute information B3) in attribute information of attribute type B is 3, and the number of mutually exclusive attribute information (i.e., attribute information C1 and attribute information C2) in attribute information of attribute type C is 2. Therefore, this Trie tree contains a total of 10 nodes excluding the root node.

[0068] like Figure 5 As shown, in the actual storage structure of the Trie tree, which serves as the sub-index of the aforementioned attributes, excluding the root node, the nodes at the first level represent attribute information of attribute type A, the nodes at the second level represent attribute type C, and the nodes at the third level represent attribute type B. Specifically, the number of mutually exclusive attribute information in attribute type A is 1, the number of mutually exclusive attribute information in attribute type C is 2, and the number of mutually exclusive attribute information in attribute type B is 3. Therefore, this Trie tree contains a total of 9 nodes excluding the root node.

[0069] Therefore, in the Trie tree that serves as the attribute sub-index, maintaining the number of attribute information corresponding to the upper level in any two adjacent levels is no greater than the number of attribute information corresponding to the lower level in those two levels. This reduces the number of nodes in the Trie tree. Consequently, the storage space occupied by the attribute sub-index can be optimized, thereby reducing the maintenance workload of the index composed of attribute sub-indexes and vector sub-indexes.

[0070] In some embodiments, the aforementioned vector sub-indexes can be built based on vector retrieval schemes such as Exact Nearest Neighbor Search (ENS) and Approximate Nearest Neighbor Search (ANNS). For example, the vector sub-indexes can specifically be HNSW (Hierarchical Navigable Small World) indexes or IVF_FLAT indexes, etc.

[0071] In some embodiments, since vector datasets are typically large, directly maintaining the vector index used to retrieve vector data in memory would require a significant amount of memory storage space. Therefore, for devices deploying the aforementioned database, the attribute sub-index can be maintained in its memory, and / or the vector sub-index can be maintained on its disk; that is, the attribute sub-index can be stored in memory, and / or the vector sub-index can be stored on disk.

[0072] In practical applications, for the devices used to deploy the above databases, their Random Access Memory (RAM) can be used as memory, that is, the above attribute sub-indexes can be stored in the RAM; Hard Disk Drive (HDD), Solid State Drive (SSD) or other forms of persistent storage devices can be used as disks, that is, the above memory sub-indexes can be stored in the Hard Disk Drive, Solid State Drive or other forms of persistent storage devices.

[0073] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating a vector hybrid retrieval method in an exemplary embodiment of this application.

[0074] In this embodiment, as described above, each data set in the database may contain vector data and a subset of attribute information; the index in the database may include an attribute sub-index built based on the subset of attribute information contained in the data sets in the database, and a vector sub-index built based on the vector data contained in the data sets in each data set group; wherein, the data set group can be obtained by grouping the data sets in the database, and the subsets of attribute information contained in the data sets in each data set group are matched; the attribute sub-index can be used to indicate the vector sub-index associated with each subset of attribute information; the vector sub-index can be used to indicate the data set containing each vector data.

[0075] The above-mentioned vector hybrid retrieval method can be applied to, for example... Figure 1 The database system shown. For example, the vector hybrid retrieval method can be performed by a device used to deploy a database management system within that database system.

[0076] like Figure 6 As shown, the above-mentioned vector hybrid retrieval method may include the following steps:

[0077] Step 602: Obtain a query request for the database; wherein the query request includes target vector data and a subset of target attribute information.

[0078] In this embodiment, when a query request for the aforementioned database is received, a vector hybrid retrieval can be performed on that database. At this time, the query request may include vector data for retrieval (which may be referred to as target vector data) and a subset of attribute information (which may be referred to as target attribute information subset).

[0079] For example, a user can initiate a data retrieval operation against the database through the aforementioned database application, and input vector data and a subset of attribute information for retrieval. The database application can construct a query request based on the vector data and the subset of attribute information, and send the query request to the aforementioned database management system, so that the database management system can respond to the query request and perform a vector hybrid retrieval in the database based on the vector data and the subset of attribute information in the query request.

[0080] Step 604: In response to the query request, based on the subset of target attribute information, search in the attribute sub-index to determine the target vector sub-index associated with the subset of target attribute information.

[0081] In this embodiment, upon receiving the aforementioned query request, a response can be provided to the query request. Specifically, based on the aforementioned subset of target attribute information in the query request, a search can be performed in the aforementioned attribute sub-index to determine the vector sub-index (which may be referred to as the target vector sub-index) associated with the subset of target attribute information.

[0082] In some embodiments, as described above, in order to reduce the maintenance workload of the index composed of attribute sub-indexes and vector sub-indexes and accelerate the final data retrieval speed, the attribute sub-index can be a Trie tree-based index; each node in the Trie tree can represent each attribute information; the database can also store the correspondence between the leaf nodes in the Trie tree and the vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree can be a vector sub-index associated with a subset of attribute information composed of nodes on the path from the root node to each leaf node in the Trie tree.

[0083] In the above scenario, a search can be performed in the Trie tree, which serves as the attribute sub-index, based on the aforementioned subset of target attribute information, to determine the path from the root node of the Trie tree to a specific leaf node (which may be referred to as the target leaf node). The subset of attribute information consisting of nodes along the path from the root node to the target leaf node is the target attribute information subset.

[0084] For example, assuming the target attribute information subset is {attribute information 1, attribute information 2, attribute information 3}, we can start from the root node in the Trie tree, which serves as the sub-index of the attribute, and search for attribute information 1 in the first level (the level below the root node); assuming the node representing attribute information 1 is node 1, we can continue searching for attribute information 2 in the second level (the level below the first level); assuming the node representing attribute information 2 is node 2, we can continue searching for attribute information 3 in the third level (the level below the second level); assuming the node representing attribute information 2 is node 3, we can determine the path as root node → node 1 → node 2 → node 3, where node 3 is the target leaf node.

[0085] Subsequently, the vector sub-index corresponding to the aforementioned target leaf node can be used to determine the target vector sub-index associated with the aforementioned subset of target attribute information.

[0086] Step 606: Based on the target vector data, search in the target vector sub-index to determine the target data set containing the target vector data, and based on the target data set, determine the query result corresponding to the query request.

[0087] In this embodiment, once the target vector sub-index is determined, the target vector data in the query request can be searched in the target vector sub-index to determine the data set containing the target vector data (which can be called the target data set). Based on the target data set, the query result corresponding to the query request can be determined (for example, the target data set can be directly output as the query result), thereby completing the vector hybrid retrieval initiated by the query request.

[0088] In some embodiments, as described above, the vector sub-index can be stored on a disk. In this case, since programs and data stored on the disk typically need to be loaded into random access memory for fast access and execution by a processor (e.g., CPU), once the target vector sub-index is determined, the target vector sub-index on the disk can be loaded into memory first, and then a search can be performed on the target vector sub-index in memory based on the target vector data.

[0089] In some embodiments, the vector hybrid retrieval method in this application can support batch query processing. Batch query processing refers to submitting multiple queries or query-related operations to the database system for centralized execution at once. This can significantly improve the performance and efficiency of the database system, especially when processing large amounts of data or frequent interactions.

[0090] However, as mentioned earlier, each time a vector hybrid retrieval is performed, the required vector sub-indexes need to be loaded from disk into memory, which constitutes a disk I / O operation. In practical applications, disk I / O operations incur significant time overhead. If a disk I / O operation is performed for each query request in a batch query process, it will obviously affect query efficiency.

[0091] Therefore, when receiving batch query requests for the aforementioned database, the batch query process can be optimized. These batch query requests can include multiple query requests belonging to the same batch.

[0092] Specifically, firstly, in response to the aforementioned batch query request, a search can be performed in the aforementioned attribute sub-index based on the subset of attribute information in each query request contained in the batch query request, in order to determine the vector sub-index associated with the subset of attribute information in each query request.

[0093] Then, the query requests in the batch query requests can be grouped according to the associated vector sub-indexes, so that query requests with the same associated vector sub-indexes are grouped into the same group.

[0094] Then, for each query request group obtained by grouping, the vector sub-index associated with the query request group on disk can be loaded into memory, and based on the vector data in each query request in the query request group, a search is performed in the vector sub-index associated with the query request group in memory to determine the data set containing the vector data in each query request in the query request group, which serves as the query result corresponding to each query request in the query request group.

[0095] Finally, the query results corresponding to each query request in the above batch query request can be summarized to determine the batch query result corresponding to the batch query request.

[0096] For example, suppose the above batch query request includes three query requests: query request 1, query request 2, and query request 3. Query request 1 includes vector data 1 and attribute information subset A, query request 2 includes vector data 2 and attribute information subset A, and query request 3 includes vector data 3 and attribute information subset B. Assuming that attribute information subset A is associated with vector sub-index A and attribute information subset B is associated with vector sub-index B, then firstly, we can search in the above attribute sub-indexes based on the attribute information subsets in these three query requests. Thus, we can determine that the vector sub-index associated with attribute information subset A in query request 1 and query request 2 is vector sub-index A, and the vector sub-index associated with attribute information subset B in query request 3 is vector sub-index B.

[0097] Then, since the vector sub-indexes associated with the subsets of attribute information in query request 1 and query request 2 are the same, query request 1 and query request 2 can be assigned to query request group 1, and query request 3 can be assigned to query request group 2.

[0098] Next, the vector sub-index A on disk can be loaded into memory. This allows a search to be performed in sub-index A based on vector data 1 from query request 1 to determine the query result corresponding to query request 1. Similarly, a search to be performed in sub-index A based on vector data 2 from query request 2 can be performed to determine the query result corresponding to query request 2. Furthermore, the vector sub-index B on disk can be loaded into memory. This allows a search to be performed in sub-index B based on vector data 3 from query request 3 to determine the query result corresponding to query request 3.

[0099] Finally, the query results corresponding to query request 1, query request 2, and query request 3 can be summarized into the batch query results corresponding to this batch query request.

[0100] By optimizing the batch query process, the number of disk I / O operations can be effectively reduced, thereby reducing the workload of batch queries in vector hybrid retrieval.

[0101] In some embodiments, as described above, in the Trie tree that serves as the attribute sub-index, nodes at the same level represent attribute information of the same attribute type. In this case, to further reduce the maintenance workload of the index composed of attribute sub-indexes and vector sub-indexes, the structure of the Trie tree can be optimized to ensure that the number of attribute information corresponding to the upper level of any two adjacent levels in the Trie tree is not greater than the number of attribute information corresponding to the lower level of those two levels.

[0102] Specifically, the number of non-matching attribute information among the attribute information represented by each level node in the above Trie tree can be counted as the attribute information count corresponding to each level. Subsequently, the attribute information counts corresponding to the two adjacent levels in the Trie tree can be compared in order from top to bottom. If the attribute information count corresponding to the upper level of these two levels is greater than the attribute information count corresponding to the lower level, then the order of these two levels can be swapped.

[0103] For example Figure 4Taking the Trie tree shown as an example, in this Trie tree, apart from the root node, the nodes located in the first level represent attribute information of attribute type A. Since the number of non-matching attribute information in attribute type A is 1, it can be determined that the number of attribute information corresponding to the first level is 1. The nodes located in the second level represent attribute type B. Since the number of non-matching attribute information in attribute type B is 3, it can be determined that the number of attribute information corresponding to the second level is 3. The nodes located in the third level represent attribute type C. Since the number of non-matching attribute information in attribute type C is 2, it can be determined that the number of attribute information corresponding to the third level is 2.

[0104] Subsequently, following the order from top to bottom, firstly, the number of attribute information corresponding to the first (upper) and second (lower) adjacent layers can be compared. Since the number of attribute information corresponding to the first layer is less than that corresponding to the second layer, the order of the first and second layers does not need to be swapped. Then, the number of attribute information corresponding to the second (upper) and third (lower) adjacent layers can be compared. Since the number of attribute information corresponding to the second layer is greater than that corresponding to the third layer, the order of the second and third layers can be swapped. That is, nodes originally located in the third layer are moved to the second layer, and nodes originally located in the second layer are moved to the third layer, while adjusting the connections between nodes. After completing the swap of the order of the second and third layers, the new Trie tree is as follows: Figure 5 As shown.

[0105] In some embodiments, the structure of the Trie tree, which serves as the attribute sub-index, can be optimized periodically at certain time intervals. Alternatively, since the attribute sub-index is typically stored in memory, the memory space occupied by the index (including the attribute sub-index, or including the attribute sub-index and the vector sub-index loaded into memory) can be checked in real time or periodically to see if it exceeds a preset threshold. If the memory space occupied by the index exceeds the threshold, the structure of the Trie tree can be optimized, that is, the number of mismatched attribute information among the attribute information represented by each layer node in the Trie tree can be counted as the number of attribute information corresponding to each layer.

[0106] In one or more embodiments of the present application, each data set in the database may contain vector data and subsets of attribute information. The index may include an attribute sub-index established based on the subsets of attribute information contained in the data set in the database, which indicates the vector sub-index associated with each subset of attribute information, and a vector sub-index established based on the vector data contained in the data set in each data set group, which indicates the data set containing each vector data, wherein the subsets of attribute information contained in the data set in each data set group match. Accordingly, when a query request for the database is obtained, the target vector sub-index may be searched first based on the target subset of attribute information in the query request to determine the target vector sub-index associated with the target subset of attribute information. Then, the target vector data in the query request may be searched in the target vector sub-index to determine the target data set containing the target vector data. Finally, the query result corresponding to the query request may be determined based on the target data set.

[0107] Using the above method, for a dataset containing vector data and attribute information subsets stored in the database, data retrieval can be performed by combining vector indexes built based on vector data and attribute indexes built based on attribute information, thereby achieving hybrid vector retrieval.

[0108] Corresponding to the embodiments of the methods described above, this application also provides embodiments of the apparatus.

[0109] Please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the structure of a device according to an exemplary embodiment of this application. At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, memory 708, and non-volatile memory 710, and may also include other necessary hardware. One or more embodiments of this application can be implemented in software, for example, the processor 702 reads the corresponding computer program from the non-volatile memory 710 into memory 708 and then runs it. Of course, besides software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic modules, but can also be hardware or logic devices.

[0110] Please refer to Figure 8 , Figure 8 This is a block diagram illustrating a vector hybrid retrieval device according to an exemplary embodiment of this application.

[0111] The above-mentioned vector hybrid retrieval device can be applied to Figure 7The device shown is used to implement the technical solution of this application. It should be noted that each data set in the database contains vector data and a subset of attribute information; the index in the database includes an attribute sub-index built based on the subset of attribute information, and a vector sub-index built based on the vector data contained in the data sets of each data set group; the subsets of attribute information contained in the data sets of each data set group are matched; the attribute sub-index is used to indicate the vector sub-index associated with each subset of attribute information; the vector sub-index is used to indicate the data set containing each vector data. The device may include:

[0112] The query acquisition module 802 acquires a query request for the database; wherein the query request includes target vector data and a subset of target attribute information;

[0113] The attribute lookup module 804, in response to the query request, performs a search in the attribute sub-index based on the target attribute information subset to determine the target vector sub-index associated with the target attribute information subset;

[0114] The vector lookup module 806 searches within the target vector sub-index based on the target vector data to determine the target data set containing the target vector data, and determines the query result corresponding to the query request based on the target data set.

[0115] In some embodiments, the attribute sub-index is a Trie tree-based index; each node in the Trie tree represents each attribute information; the database also stores the correspondence between the leaf nodes in the Trie tree and the vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree is a vector sub-index associated with a subset of attribute information composed of nodes on the path from the root node to each leaf node in the Trie tree;

[0116] The step of searching within the attribute sub-index based on the subset of target attribute information to determine the target vector sub-index associated with the subset of target attribute information includes:

[0117] Based on the target attribute information subset, a search is performed in the Trie tree to determine the path from the root node to the target leaf node; wherein, the attribute information subset consisting of the nodes on the path is the target attribute information subset;

[0118] The vector sub-index corresponding to the target leaf node is determined as the target vector sub-index associated with the subset of target attribute information.

[0119] In some embodiments, nodes at the same level in the Trie tree represent attribute information with the same attribute type;

[0120] The device further includes:

[0121] The statistics module counts the number of mismatched attribute information among the attribute information represented by each layer node in the Trie tree, and uses this as the number of attribute information corresponding to each layer.

[0122] The comparison module compares the number of attribute information corresponding to the two adjacent layers in the Trie tree in order from top to bottom.

[0123] The optimization module, if the number of attribute information corresponding to the upper layer of the two layers is greater than the number of attribute information corresponding to the lower layer of the two layers, then swaps the upper and lower layer order of the two layers.

[0124] In some embodiments, the attribute sub-index is stored in memory;

[0125] The device further includes:

[0126] The detection module detects whether the memory space occupied by the index exceeds a preset threshold.

[0127] The statistics module performs the following specific actions:

[0128] If the memory space occupied by the index exceeds the threshold, the number of mismatched attribute information among the attribute information represented by each layer node in the Trie tree is counted as the number of attribute information corresponding to each layer.

[0129] In some embodiments, the vector sub-index is stored on disk;

[0130] The search based on the target vector data within the target vector sub-index includes:

[0131] The target vector sub-index is loaded from the disk into memory, and a search is performed in the target vector sub-index in memory based on the target vector data.

[0132] In some embodiments, the apparatus further includes:

[0133] A batch query acquisition module acquires batch query requests for the database; wherein, the batch query requests include multiple query requests belonging to the same batch;

[0134] The batch processing attribute lookup module, in response to the batch processing query request, searches in the attribute sub-index based on the attribute information subset in each query request to determine the vector sub-index associated with the attribute information subset in each query request;

[0135] The query grouping module groups the query requests in the batch query request according to the associated vector sub-index, so as to group query requests with the same associated vector sub-index into the same group;

[0136] The batch vector lookup module loads the vector sub-index associated with each query request group from the disk into memory, and searches in the vector sub-index associated with the query request group in memory based on the vector data in the query requests in the query request group, so as to determine the data set containing the vector data in each query request in the query request group, which is used as the query result corresponding to each query request in the query request group.

[0137] The result processing module summarizes the query results corresponding to each query request in the batch query request to determine the batch query result corresponding to the batch query request.

[0138] For the device embodiments, they basically correspond to the method embodiments; therefore, relevant details can be found in the descriptions of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the technical solution of this application according to actual needs.

[0139] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0140] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0142] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0143] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0145] The terminology used in one or more embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this application. The singular forms “a,” “the,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. The term “and / or” refers to and includes any or all possible combinations of one or more associated listed items.

[0146] The terms "an embodiment," "some embodiments," "example," "specific example," or "one implementation," as used in one or more embodiments of this application, refer to specific features or characteristics described in connection with that embodiment, which are included in at least one embodiment of this application. Illustrative descriptions of these terms do not necessarily refer to the same embodiment. Furthermore, the described specific features or characteristics may be combined in a suitable manner in one or more embodiments of this application. Moreover, different embodiments and specific features or characteristics from different embodiments may be combined without contradiction.

[0147] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this application, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0148] The above description is merely a preferred embodiment of one or more embodiments of this application and is not intended to limit the scope of one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the protection scope of one or more embodiments of this application.

[0149] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A database system, the database system comprising a database; wherein: Each data set in the database contains vector data and a subset of attribute information; The indexes in the database include attribute sub-indexes based on the attribute information subsets and vector sub-indexes based on the vector data contained in the data sets of each data set group; the attribute information subsets contained in the data sets of each data set group are matched. Each node in the Trie tree represents a different attribute information; nodes at the same level in the Trie tree represent attribute information of the same attribute type, and nodes at different levels represent attribute information of different attribute types; the database also stores the correspondence between the leaf nodes in the Trie tree and the vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree is a vector sub-index associated with a subset of attribute information formed by nodes on the path from the root node to each leaf node in the Trie tree; the vector sub-index is used to indicate a data set containing each vector data.

2. In the database system according to claim 1, the number of attribute information corresponding to the upper layer of two adjacent layers in the Trie tree is not greater than the number of attribute information corresponding to the lower layer of the two layers; the number of attribute information corresponding to each layer in the Trie tree is the number of attribute information that do not match each other among the attribute information represented by the nodes of each layer.

3. The database system according to claim 2, wherein the attribute sub-index is stored in memory; and / or, the vector sub-index is stored on disk.

4. A vector hybrid retrieval method; each data set in the database contains vector data and a subset of attribute information; the index in the database includes an attribute sub-index based on a Trie tree built on the subset of attribute information, and a vector sub-index built on the vector data contained in the data sets of each data set group; the subsets of attribute information contained in the data sets of each data set group are matched; each node in the Trie tree represents a different attribute information; nodes at the same level in the Trie tree represent attribute information of the same attribute type, and nodes at different levels represent attribute information of different attribute types; the database also stores the correspondence between the leaf nodes in the Trie tree and the vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree is a vector sub-index associated with the subset of attribute information formed by the nodes on the path from the root node to each leaf node in the Trie tree; The vector sub-index is used to indicate a data set containing individual vector data; The method includes: Obtain a query request for the database; wherein the query request includes target vector data and a subset of target attribute information; In response to the query request, a search is performed in the Trie tree based on the target attribute information subset to determine the path from the root node to the target leaf node; wherein the attribute information subset composed of the nodes on the path is the target attribute information subset; and the vector sub-index corresponding to the target leaf node is determined as the target vector sub-index associated with the target attribute information subset. Based on the target vector data, a search is performed in the target vector sub-index to determine the target data set containing the target vector data, and based on the target data set, the query result corresponding to the query request is determined.

5. The method according to claim 4, further comprising: The number of mismatched attribute information among the attribute information represented by each layer node in the Trie tree is counted as the number of attribute information corresponding to each layer. In order from top to bottom, the number of attribute information corresponding to the two adjacent layers in the Trie tree is compared in turn. If the number of attribute information corresponding to the upper layer of the two layers is greater than the number of attribute information corresponding to the lower layer of the two layers, then the upper and lower layer order of the two layers is swapped.

6. The method according to claim 5, wherein the attribute sub-index is stored in memory; Before counting the number of mismatched attribute information among the attribute information represented by nodes at each level in the Trie tree, as the number of attribute information corresponding to each level, the method further includes: Detect whether the memory space occupied by the index exceeds a preset threshold; If the memory space occupied by the index exceeds the threshold, the number of mismatched attribute information among the attribute information represented by each layer node in the Trie tree is counted as the number of attribute information corresponding to each layer.

7. The method according to claim 4, wherein the vector sub-index is stored on disk; The search based on the target vector data within the target vector sub-index includes: The target vector sub-index is loaded from the disk into memory, and a search is performed in the target vector sub-index in memory based on the target vector data.

8. The method according to claim 7, further comprising: Obtain batch query requests for the database; wherein the batch query requests include multiple query requests belonging to the same batch; In response to the batch query request, a search is performed in the attribute sub-index based on the subset of attribute information in each query request to determine the vector sub-index associated with the subset of attribute information in each query request; The query requests in the batch query requests are grouped according to the associated vector sub-indexes, so that query requests with the same associated vector sub-indexes are grouped into the same group; The vector sub-index associated with each query request group in the disk is loaded into memory, and based on the vector data in the query requests in the query request group, a search is performed in the vector sub-index associated with the query request group in memory to determine the data set containing the vector data in each query request in the query request group, which is used as the query result corresponding to each query request in the query request group. The query results corresponding to each query request in the batch query request are summarized to determine the batch query result corresponding to the batch query request.

9. A vector hybrid retrieval device; each data set in the database contains vector data and a subset of attribute information; the index in the database includes an attribute sub-index based on a Trie tree built on the subset of attribute information, and a vector sub-index built on the vector data contained in the data sets of each data set group; the subsets of attribute information contained in the data sets of each data set group are matched; each node in the Trie tree represents a subset of attribute information; nodes at the same level in the Trie tree represent attribute information of the same attribute type, and nodes at different levels represent attribute information of different attribute types; the database also stores the correspondence between the leaf nodes in the Trie tree and the vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree is a vector sub-index associated with the subset of attribute information formed by the nodes on the path from the root node to each leaf node in the Trie tree; The vector sub-index is used to indicate a data set containing individual vector data; The device includes: The query acquisition module acquires a query request for the database; wherein the query request includes target vector data and a subset of target attribute information; The attribute lookup module, in response to the query request, performs a search in the Trie tree based on the target attribute information subset to determine the path from the root node to the target leaf node; wherein the attribute information subset composed of the nodes on the path is the target attribute information subset; and the vector sub-index corresponding to the target leaf node is determined as the target vector sub-index associated with the target attribute information subset. The vector lookup module searches within the target vector sub-index based on the target vector data to determine the target data set containing the target vector data, and then determines the query result corresponding to the query request based on the target data set.

10. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 4 to 8 by executing the executable instructions.

11. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 4 to 8.

Images