Database system and vector hybrid retrieval method

Abstract

One or more embodiments of the invention provide a database system and a vector hybrid retrieval method, an index in a database comprises an attribute sub-index which is established based on an attribute information subset contained in a data set in the database and is used for indicating a vector sub-index, the vector sub-index is established based on vector data contained in the data set in each data set group and is used for indicating the data set; the attribute information subsets contained in the data sets in the data set groups are matched; the method comprises the following steps: acquiring a query request comprising target vector data and a target attribute information subset; in response to the query request, searching in the attribute sub-index based on the target attribute information subset to determine a target vector sub-index associated with the target attribute information subset; and based on the target vector data, searching in the target vector sub-index to determine a target data set containing the target vector data, and based on the target data set, determining a query result corresponding to the query request.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of database technology, and in particular, to a database system and a vector hybrid retrieval method. Background Art

[0002] With the rapid development of artificial intelligence (AI), data representation is no longer limited to traditional structured data. An increasing amount of data is presented in unstructured forms such as images, videos, and audio. Structured data, on the other hand, is typically stored in relational databases as two-dimensional tables. To effectively process and analyze unstructured data, embedding is often used to convert unstructured data into vector form. This involves vectorizing the unstructured data to facilitate subsequent similarity calculations and retrieval operations. Embedding refers to the process of mapping high-dimensional, sparse data into a low-dimensional, dense vector space, effectively expressing the data's inherent structural characteristics. Vector similarity retrieval involves calculating the distance between vectors (such as Euclidean distance and cosine similarity). The smaller the distance between vectors, the higher the similarity.

[0003] In practical applications, single-vector similarity retrieval can only perform similarity retrieval on unstructured data, ignoring the structured attribute information that accompanies unstructured data. For example, in e-commerce recommendation systems, vector similarity retrieval can only retrieve products that are most visually similar to the current product image, but ignores attribute information such as brand, price, and category. Due to the lack of effective utilization of this structured attribute information, single-vector similarity retrieval often fails to meet the diverse retrieval needs of complex service scenarios. Therefore, it is desirable to be able to combine vector similarity retrieval with structured attribute information for filtering. Summary of the Invention

[0004] One or more embodiments of the present application provide the following technical solutions: The present application provides a database system, which includes a database; wherein: Each data set in the database includes vector data and attribute information subsets; The index in the database includes an attribute sub-index established based on the attribute information subset and a vector sub-index established based on the vector data contained in the data sets in each data set group; the attribute information subset contained in the data sets in each data set group matches; The attribute sub-index is used to indicate a vector sub-index associated with each attribute information subset; the vector sub-index is used to indicate a data set containing each vector data.

[0005] The present application also provides a vector hybrid retrieval method; each data set in a database includes vector data and attribute information subsets; the index in the database includes an attribute sub-index established based on the attribute information subsets, and a vector sub-index established based on the vector data contained in the data sets in each data set group; the attribute information subsets contained in the data sets in each data set group are matched; the attribute sub-index is used to indicate the vector sub-index associated with each attribute information subset; the vector sub-index is used to indicate the data set containing each vector data; The method comprises: Obtaining a query request for the database; wherein the query request includes target vector data and a subset of target attribute information; In response to the query request, searching in the attribute sub-index based on the target attribute information subset to determine a target vector sub-index associated with the target attribute information subset; Based on the target vector data, a search is performed in the target vector sub-index to determine a target data set containing the target vector data, and based on the target data set, a query result corresponding to the query request is determined.

[0006] The present application also provides a vector hybrid retrieval device; each data set in a database includes vector data and attribute information subsets; the index in the database includes an attribute sub-index established based on the attribute information subsets, and a vector sub-index established based on the vector data contained in the data sets in each data set group; the attribute information subsets contained in the data sets in each data set group are matched; the attribute sub-index is used to indicate the vector sub-index associated with each attribute information subset; the vector sub-index is used to indicate the data set containing each vector data; The device comprises: A query acquisition module, which acquires a query request for the database; wherein the query request includes target vector data and a target attribute information subset; an attribute search module, in response to the query request, searching in the attribute sub-index based on the target attribute information subset to determine a target vector sub-index associated with the target attribute information subset; The vector search module searches the target vector sub-index based on the target vector data to determine a target data set containing the target vector data, and determines a query result corresponding to the query request based on the target data set.

[0007] The present application also provides an electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the steps of any of the above methods by running the executable instructions.

[0008] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0009] In the above technical solution, each data set in the database may contain vector data and attribute information subsets, and the index may include an attribute sub-index established based on the attribute information subsets contained in the data sets in the database, for indicating the vector sub-index associated with each attribute information subset, and a vector sub-index established based on the vector data contained in the data sets in each data set group, for indicating the data sets containing each vector data, wherein the attribute information subsets contained in the data sets in each data set group are matched; accordingly, when a query request for the database is obtained, the attribute sub-index can be searched first based on the target attribute information subset in the query request to determine the target vector sub-index associated with the target attribute information subset, and then the target vector data in the query request can be searched in the target vector sub-index to determine the target data set containing the target vector data, and the query result corresponding to the query request can be determined based on the target data set.

[0010] Using the above method, for a data set containing vector data and attribute information subsets stored in a database, data retrieval can be performed by combining vector indexes established based on vector data and attribute indexes established based on attribute information, thereby realizing vector mixed retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The following is a description of the accompanying drawings required for describing the exemplary embodiments, in which: Figure 1 It is a schematic diagram of the architecture of a database system shown in an exemplary embodiment of the present application.

[0012] Figure 2 It is a schematic diagram of the structure of a database index shown in an exemplary embodiment of the present application.

[0013] Figure 3 This is a schematic diagram of the structure of an attribute sub-index shown in an exemplary embodiment of the present application.

[0014] Figure 4 This is a schematic diagram of the structure of another attribute sub-index shown in an exemplary embodiment of the present application.

[0015] Figure 5 This is a schematic diagram of the structure of another attribute sub-index shown in an exemplary embodiment of the present application.

[0016] Figure 6 This is a flowchart of a vector hybrid retrieval method shown in an exemplary embodiment of the present application.

[0017] Figure 7 It is a structural diagram of a device shown in an exemplary embodiment of the present application.

[0018] Figure 8 It is a block diagram of a vector hybrid retrieval device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0019] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. Rather, they are merely examples consistent with certain aspects of one or more embodiments of the present application.

[0020] It should be noted that, in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the method may include more or fewer steps than those described in this application. In addition, a single step described in this application may be broken down into multiple steps for description in other embodiments; and multiple steps described in this application may be combined into a single step for description in other embodiments.

[0021] This type of vector retrieval, combining structured attribute information with vector similarity retrieval, is called vector hybrid retrieval. In other words, vector hybrid retrieval involves the simultaneous retrieval of both unstructured data (the vector representation of the original data) and structured data (the attribute information of the original data).

[0022] Vector hybrid retrieval can not only improve the relevance and accuracy of retrieval results, but also enhance the flexibility and practicality of the system, meeting the actual needs of multi-dimensional data fusion retrieval.

[0023] One or more embodiments of the present application provide a technical solution for implementing vector hybrid retrieval, in which each data set in a database may contain vector data and attribute information subsets, and the index may include an attribute sub-index established based on the attribute information subsets contained in the data sets in the database, for indicating the vector sub-index associated with each attribute information subset, and a vector sub-index established based on the vector data contained in the data sets in each data set group, for indicating the data sets containing each vector data, wherein the attribute information subsets contained in the data sets in each data set group match; accordingly, when a query request for the database is obtained, a search can be first performed in the attribute sub-index based on the target attribute information subset in the query request to determine the target vector sub-index associated with the target attribute information subset, and then a search can be performed in the target vector sub-index based on the target vector data in the query request to determine the target data set containing the target vector data, and a query result corresponding to the query request can be determined based on the target data set.

[0024] Using the above method, for a data set containing vector data and attribute information subsets stored in a database, data retrieval can be performed by combining vector indexes established based on vector data and attribute indexes established based on attribute information, thereby realizing vector mixed retrieval.

[0025] Please refer to Figure 1 , Figure 1 It is a schematic diagram of the architecture of a database system shown in an exemplary embodiment of the present application.

[0026] A database system is a software system used to store, retrieve and manage data. Figure 1 As shown, the database system may include at least a database, a database management system and a database application.

[0027] A database is a collection of stored data. Data is organized in a database according to a specific schema or model for easy retrieval and management.

[0028] A database management system (DBMS) is a software system that serves as the core component for controlling and managing databases. It is used to define, create, maintain, and manipulate databases. It provides a range of tools and interfaces for managing databases, such as those for create, read, update, and delete operations. DBMSs primarily control the storage, retrieval, update, and management of data, while also ensuring data accuracy, consistency, security, and backup and recovery capabilities.

[0029] A database application is a front-end application that interacts with a database to implement specific service logic and functionality. It provides a user-facing interface, which can be a graphical user interface, command line interface, or other form of interface.

[0030] In actual applications, users can initiate specific operations on the database (such as index creation operations, data retrieval operations, data query operations, etc.) through database applications, and the database management system can perform corresponding operations on the database.

[0031] In the context of databases, querying generally refers to executing specific database commands to retrieve data, searching generally refers to attempting to locate specific data, and retrieval covers the broader process of obtaining and possibly processing data.

[0032] The aforementioned database and database management system can be deployed on a server. The server can be a single physical server or a server cluster consisting of multiple independent physical hosts. Alternatively, the server can be a virtual server or cloud server hosted by a host cluster. The aforementioned database application can be deployed on a terminal device connected to the server through a network (including various types of wired or wireless networks). The terminal device can be a personal computer (PC), desktop computer, laptop computer, tablet computer, smartphone, or personal digital assistant (PDA).

[0033] In the database system described in one or more embodiments of the present application, the data stored in the database may be in the form of a data set. Specifically, a data set may include a vector data and an attribute information set (referred to as an attribute information subset to distinguish it from the data set). The attribute information subset may include one or more attribute information corresponding to the vector data.

[0034] For example, in an e-commerce scenario, the data stored in the database could be product-related data. For a particular product, the data could be processed into a data set. This data set might include vector data corresponding to the product image (e.g., the embedding vector corresponding to the image) and a subset of attribute information consisting of the product's attributes (e.g., color, price, brand, etc.).

[0035] For example, in a music playback scenario, the data stored in the database could be song-related data. For a single song, the data associated with the song can be processed into a data set. This data set might include vector data corresponding to the song's audio (e.g., the embedding vector corresponding to the image), as well as a subset of attribute information consisting of the song's attributes (e.g., artist information, album information, number of likes, number of comments, etc.).

[0036] In a database system, indexes can be introduced to speed up data queries. Indexes are typically created, managed, and used by the database management system, but are stored in the database. An index is a data structure, similar to the table of contents of a book, that helps the database quickly locate the required data without having to perform a full scan. Using indexes can significantly reduce the amount of data that needs to be scanned for queries, speeding up queries, especially when processing large amounts of data. However, indexes require additional storage space, and additional time is required to maintain the index when inserting, updating, or deleting. Therefore, when designing a database, it is necessary to weigh the query performance improvements brought by indexes against their maintenance costs.

[0037] For the data in the form of data sets stored in the above-mentioned database, since a data set can contain a vector data and a subset of attribute information, a vector index can be established based on the vector data, and an attribute index can be established based on the attribute information. The vector index and the attribute index can be combined to perform data retrieval, thereby realizing vector hybrid retrieval.

[0038] Please refer to Figure 2 , Figure 2 It is a schematic diagram of the structure of a database index shown in an exemplary embodiment of the present application.

[0039] In this embodiment, as mentioned above, the data stored in the database may be in the form of a data set, wherein a data set may specifically include a vector data and an attribute information subset, and the attribute information subset may include one or more attribute information corresponding to the vector data.

[0040] Accordingly, an index can be established based on the vector data contained in the data set in the above database as a vector index, and an index can be established based on the attribute information subset contained in the data set in the database as an attribute index. Figure 2 As shown, the established vector index can be called a vector sub-index, and the established attribute index can be called an attribute sub-index. The vector sub-index and the attribute sub-index together constitute an index for implementing vector mixed retrieval and are stored in the database.

[0041] It should be noted that when establishing the above-mentioned vector sub-index based on the vector data contained in the data sets in the above-mentioned database, the data sets in the database can be first grouped according to the attribute information subsets contained therein, so that for any data set group among all data set groups obtained by grouping, the attribute information subsets contained in the data sets in the data set group match. The matching of attribute information subsets contained in two data sets can be that corresponding attribute information in the attribute information subsets contained in the two data sets match (for example, the two attribute information are the same, or the similarity between the two attribute information is greater than a specific threshold). For example, suppose the database stores three data sets (data sets are represented by "{}"), namely data set 1 = {vector = vector 1, attributes = {color = "red", price = 100, brand = "Brand X"}}, data set 2 = {vector = vector 2, attributes = {color = "red", price = 100, brand = "Brand X"}}, and data set 3 = {vector = vector 3, attributes = {color = "black", price = 200, brand = "Brand X"}}. Since the attribute information in the attribute information subset contained in data set 1 is consistent with the attribute information in the attribute information subset contained in data set 2, but is significantly different from the attribute information in the attribute information subset contained in data set 3, data sets 1 and data set 2 can be divided into one data set group, and data set 3 can be divided into another data set group. Subsequently, a vector sub-index can be established based on the vector data contained in the data sets in each data set group, that is, a vector sub-index can be established for each data set group.

[0042] In order to ensure that the above-mentioned vector sub-index and the above-mentioned attribute sub-index can be used in combination during the data retrieval process to realize vector mixed retrieval, the above-mentioned attribute sub-index can be used to indicate the vector sub-index associated with the attribute information subset contained in each data set in the above-mentioned database; that is, assuming that attribute information subset i is the attribute information subset contained in data set i in the data set library, then using this attribute sub-index, the vector sub-index associated with attribute information subset i can be quickly located (that is, the vector sub-index established based on the vector data contained in the data set in the data set group where data set i is located). The above-mentioned vector sub-index can be used to indicate the data set containing each vector data; that is, assuming that vector data j is the vector data contained in data set j in the data set library, then using the vector sub-index associated with the attribute information subset contained in data set j, the data set containing vector data j (that is, data set j) can be quickly located.

[0043] When searching for data, you can first use the attribute sub-index described above based on the attribute information subset in the query request to find the vector sub-index associated with that attribute information subset. Then, based on the vector data in the query request, use the vector sub-index to find the data set containing that vector data. This allows for vector hybrid search, meaning that both vector data and attribute information can be retrieved simultaneously.

[0044] In some embodiments, in order to optimize the storage space occupied by the above-mentioned attribute sub-index and speed up the retrieval of attribute information, thereby reducing the maintenance workload of the index composed of the attribute sub-index and the vector sub-index, and speeding up the final data retrieval speed, the structure and concept of the Trie tree (also known as the prefix tree or dictionary tree) can be used to establish the attribute sub-index.

[0045] The original Trie tree is a tree-like data structure used to efficiently store and retrieve a collection of strings. Its basic structure is that each node represents a character, and the nodes along the path from the root node to a particular node form a string. The core concept is that strings sharing a common prefix share the same path. Therefore, the original Trie tree has the following characteristics: the root node does not contain any characters; it is the entry point of the tree; each child node under the root node represents a character; the path from the root node to a particular node forms a string; and the node representing the last character of a string can be a leaf node, meaning that a leaf node has no children.

[0046] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of an attribute sub-index shown in an exemplary embodiment of the present application.

[0047] like Figure 3 As shown, the above-mentioned attribute sub-index can be an index based on a Trie tree. In the Trie tree serving as the attribute sub-index, a node can represent an attribute information; the root node does not contain attribute information, and it is the query entry. Therefore, in the Trie tree, the nodes on the path from the root node to a certain node can form an attribute information set, and the attribute information set can be a subset of the attribute information contained in a certain data set. For example, assuming that the path from the root node to the node representing attribute information N2 is root node → node representing attribute information 12 → node representing attribute information 24 → node representing attribute information 33 → node representing attribute information N2, then the attribute information subset formed by the nodes on the path is {attribute information 12, attribute information 24, attribute information 33, attribute information N2}.

[0048] Furthermore, in the Trie tree serving as the attribute sub-index, a leaf node can represent the last attribute information in an attribute information subset. Therefore, for any leaf node in the Trie tree, the leaf node can be associated with a vector sub-index associated with the attribute information subset consisting of the nodes on the path from the root node in the Trie tree to the leaf node. That is, the correspondence between the leaf node and the vector sub-index can also be stored in the database, so that when the nodes constituting the attribute information subset in the query request are found in the Trie tree, the vector sub-index corresponding to the leaf node on the path formed by these nodes can be determined as the vector sub-index associated with the attribute information subset.

[0049] In some embodiments, to further optimize the storage space occupied by the attribute sub-index, when establishing the attribute sub-index based on a Trie tree, nodes representing attribute information of the same attribute type can be placed on the same layer in the Trie tree. Attribute type refers to the classification of an attribute, and attribute information refers to the specific content of the attribute. For example, in an e-commerce scenario, attribute types can be color, price, brand, etc., and attribute information can be red (attribute type: color), 100 (attribute type: price), and brand X (attribute type: brand). In the Trie tree serving as the attribute sub-index, for any two adjacent layers, the number of attribute information corresponding to the upper layer of the two layers is no greater than the number of attribute information corresponding to the lower layer of the two layers. The number of attribute information corresponding to a layer can be the number of mutually unmatched attribute information within the attribute information represented by the nodes at that layer (i.e., the nodes located at that layer). For example, in an e-commerce scenario, assuming there are three nodes at that layer, representing the attribute information "red," "red," and "black," the number of mutually unmatched attribute information within the attribute information represented by the nodes at that layer is two.

[0050] Please refer to Figure 4 and Figure 5 , Figure 4 and Figure 5 They are respectively schematic diagrams of the structure of another attribute sub-index shown in an exemplary embodiment of the present application.

[0051] like Figure 4As shown, in the actual storage structure of the Trie tree serving as the attribute sub-index, excluding the root node, the nodes on the first level represent attribute information of attribute type A, the nodes on the second level represent attribute information of attribute type B, and the nodes on the third level represent attribute information of attribute type C. The number of mutually unmatched attribute information in the attribute information of attribute type A (i.e., attribute information A1) is 1, the number of mutually unmatched attribute information in the attribute information of attribute type B (i.e., attribute information B1, attribute information B2, and attribute information B3) is 3, and the number of mutually unmatched attribute information in the attribute information of attribute type C (i.e., attribute information C1 and attribute information C2) is 2. In this case, the Trie tree contains a total of 10 nodes, excluding the root node.

[0052] like Figure 5 As shown, in the actual storage structure of the Trie tree serving as the attribute sub-index, excluding the root node, the nodes on the first level represent attribute information of attribute type A, the nodes on the second level represent attribute information of attribute type C, and the nodes on the third level represent attribute information of attribute type B. The number of mutually unmatched attribute information in the attribute information of attribute type A is 1, the number of mutually unmatched attribute information in the attribute information of attribute type C is 2, and the number of mutually unmatched attribute information in the attribute information of attribute type B is 3. In this case, the Trie tree contains a total of 9 nodes, excluding the root node.

[0053] As can be seen from this, in the Trie tree used as the attribute sub-index, keeping the number of attribute information corresponding to the upper layer of any two adjacent layers no greater than the number of attribute information corresponding to the lower layer of those two layers can reduce the number of nodes in the Trie tree. Therefore, the storage space occupied by the attribute sub-index can be optimized, thereby reducing the maintenance workload of the index composed of the attribute sub-index and the vector sub-index.

[0054] In some embodiments, the vector sub-index can be established based on vector retrieval schemes such as Exact Nearest Neighbor Search (ENS) and Approximate Nearest Neighbor Search (ANNS). For example, the vector sub-index can be a Hierarchical Navigable Small World (HNSW) index or an IVF_FLAT index.

[0055] In some embodiments, since vector datasets are typically large, directly maintaining the vector index used to retrieve vector data in memory requires a large amount of memory storage space. Therefore, for a device used to deploy the above-mentioned database, the above-mentioned attribute sub-index can be maintained in its memory and / or the above-mentioned vector sub-index can be maintained on its disk; that is, the attribute sub-index can be stored in memory and / or the vector sub-index can be stored on disk.

[0056] In practical applications, for a device used to deploy the above-mentioned database, its random access memory (RAM) can be used as memory, that is, the above-mentioned attribute sub-index can be stored in the random access memory; a hard disk drive (HDD), a solid state drive (SSD), or other forms of persistent storage devices can be used as a disk, that is, the above-mentioned memory sub-index can be stored in a hard disk drive, a solid state drive, or other forms of persistent storage devices.

[0057] Please refer to Figure 6 , Figure 6 This is a flowchart of a vector hybrid retrieval method shown in an exemplary embodiment of the present application.

[0058] In this embodiment, as described above, each data set in the database may contain vector data and attribute information subsets; the index in the database may include an attribute sub-index established based on the attribute information subset contained in the data set in the database, and a vector sub-index established based on the vector data contained in the data set in each data set group; wherein, the data set group can be obtained by grouping the data sets in the database, and the attribute information subsets contained in the data sets in each data set group are matched; the attribute sub-index can be used to indicate the vector sub-index associated with each attribute information subset; the vector sub-index can be used to indicate the data set containing each vector data.

[0059] The above vector hybrid retrieval method can be applied to Figure 1 For example, the vector hybrid retrieval method can be executed by a device for deploying a database management system in the database system.

[0060] like Figure 6 As shown, the above-mentioned vector hybrid retrieval method may include the following steps: Step 602: Obtain a query request for the database; wherein the query request includes target vector data and a target attribute information subset.

[0061] In this embodiment, upon receiving a query request for the aforementioned database, a vector hybrid search can be performed within the database. In this case, the query request can include the vector data to be searched (referred to as the target vector data) and a subset of attribute information (referred to as the target attribute information subset).

[0062] For example, a user may initiate a data retrieval operation against the database through the database application and input vector data and a subset of attribute information to be retrieved. The database application may construct a query request based on the vector data and the subset of attribute information and send the query request to the database management system. The database management system may then respond to the query request and perform a vector hybrid search within the database based on the vector data and the subset of attribute information in the query request.

[0063] Step 604: In response to the query request, based on the target attribute information subset, search in the attribute sub-index to determine a target vector sub-index associated with the target attribute information subset.

[0064] In this embodiment, upon receiving the query request, a response may be provided. Specifically, based on the target attribute information subset in the query request, a search may be performed in the attribute sub-index to determine a vector sub-index associated with the target attribute information subset (referred to as a target vector sub-index).

[0065] In some embodiments, as described above, in order to reduce the maintenance workload of the index composed of attribute sub-indexes and vector sub-indexes and to speed up the final data retrieval speed, the above-mentioned attribute sub-index can be an index based on a Trie tree; each node in the Trie tree can represent each attribute information; the above-mentioned database can also store the correspondence between the leaf nodes in the Trie tree and the above-mentioned vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree can be a vector sub-index associated with a subset of attribute information composed of nodes on the path from the root node in the Trie tree to each leaf node.

[0066] In this case, based on the target attribute information subset, a search can be performed in the Trie tree, which serves as the attribute sub-index, to determine the path from the root node in the Trie tree to a specific leaf node (referred to as the target leaf node). The attribute information subset consisting of the nodes on the path from the root node in the Trie tree to the target leaf node is the target attribute information subset.

[0067] For example, assuming that the above-mentioned target attribute information subset is {attribute information 1, attribute information 2, attribute information 3}, you can start from the root node in the above-mentioned Trie tree as the above-mentioned attribute sub-index, and search for attribute information 1 in the first layer except the root node (that is, the layer below the root node); assuming that the node representing attribute information 1 is node 1, you can continue to search for attribute information 2 in the second layer (that is, the layer below the first layer); assuming that the node representing attribute information 2 is node 2, you can continue to search for attribute information 3 in the third layer (that is, the layer below the second layer); assuming that the node representing attribute information 2 is node 3, the path can be determined to be root node → node 1 → node 2 → node 3, and node 3 is the above-mentioned target leaf node.

[0068] Subsequently, the vector sub-index corresponding to the target leaf node may be used to determine the target vector sub-index associated with the target attribute information subset.

[0069] Step 606: Based on the target vector data, search in the target vector sub-index to determine a target data set containing the target vector data, and determine a query result corresponding to the query request based on the target data set.

[0070] In this embodiment, once the target vector sub-index is determined, a search can be performed in the target vector sub-index based on the target vector data in the query request to determine a data set containing the target vector data (which may be referred to as a target data set), and based on the target data set, a query result corresponding to the query request is determined (for example, the target data set may be directly output as the query result), thereby completing the vector hybrid retrieval initiated by the query request.

[0071] In some embodiments, as described above, the vector sub-index may be stored on disk. In this case, since programs and data stored on disk typically need to be loaded into random access memory (RAM) for rapid access and execution by a processor (e.g., a CPU), once the target vector sub-index is determined, the target vector sub-index from disk can be loaded into memory, and then the target vector data can be searched within the target vector sub-index in memory.

[0072] In some embodiments, the vector hybrid retrieval method of this application can support batch query processing. Batch query processing refers to submitting multiple queries or query-related operations to the database system at once for centralized execution. This can significantly improve the performance and efficiency of the database system, especially when processing large amounts of data or frequent interactions.

[0073] However, as mentioned earlier, each time a vector hybrid search is performed, the required vector sub-indexes must be loaded from disk into memory, which is a disk I / O operation. In practical applications, disk I / O operations incur significant overhead. If disk I / O is performed for each query request in a batch query, query efficiency will be significantly reduced.

[0074] Therefore, when a batch query request for the above database is obtained, the batch query process can be optimized. The batch query request may include multiple query requests belonging to the same batch.

[0075] Specifically, first, in response to the batch query request, based on the attribute information subsets in each query request contained in the batch query request, a search can be performed in the attribute sub-index to determine the vector sub-index associated with the attribute information subsets in each query request.

[0076] Then, the query requests in the batch query requests may be grouped according to the associated vector sub-indexes, so that query requests with the same associated vector sub-indexes are divided into the same group.

[0077] Then, for each query request group obtained by grouping, the vector sub-index associated with the query request group in the disk can be loaded into the memory, and based on the vector data in each query request in the query request group, the vector sub-index associated with the query request group in the memory can be searched to determine the data set containing the vector data in each query request in the query request group as the query result corresponding to each query request in the query request group.

[0078] Finally, the query results corresponding to the respective query requests in the batch query request may be summarized to determine the batch query result corresponding to the batch query request.

[0079] For example, assuming that the above-mentioned batch query request includes three query requests, namely query request 1, query request 2, and query request 3, wherein query request 1 includes vector data 1 and attribute information subset A, query request 2 includes vector data 2 and attribute information subset A, and query request 3 includes vector data 3 and attribute information subset B, and assuming that attribute information subset A is associated with vector sub-index A, and attribute information subset B is associated with vector sub-index B, then first, based on the attribute information subsets in these three query requests, respectively, a search can be performed in the above-mentioned attribute sub-index, thereby determining that the vector sub-index associated with attribute information subset A in query request 1 and query request 2 is vector sub-index A, and the vector sub-index associated with attribute information subset B in query request 3 is vector sub-index B.

[0080] Then, since the vector sub-index associated with the attribute information subsets in query request 1 and query request 2 is the same, query request 1 and query request 2 can be divided into query request group 1, and query request 3 can be divided into query request group 2.

[0081] Then, vector sub-index A on disk can be loaded into memory, so that based on vector data 1 in query request 1, vector sub-index A in memory can be searched to determine the query result corresponding to query request 1. Based on vector data 2 in query request 2, vector sub-index A in memory can be searched to determine the query result corresponding to query request 2. In addition, vector sub-index B on disk can be loaded into memory, so that based on vector data 3 in query request 3, vector sub-index B in memory can be searched to determine the query result corresponding to query request 3.

[0082] Finally, the query result corresponding to query request 1, the query result corresponding to query request 2, and the query result corresponding to query request 3 may be aggregated into a batch query result corresponding to the batch query request.

[0083] By optimizing the batch query process, the number of disk IO operations can be effectively reduced, thereby reducing the batch query workload in vector hybrid retrieval.

[0084] In some embodiments, as previously described, in the Trie tree serving as the attribute sub-index, nodes at the same level represent attribute information of the same attribute type. In this case, to further reduce the maintenance workload of the index composed of the attribute sub-index and the vector sub-index, the structure of the Trie tree can be optimized to ensure that the number of attribute information corresponding to the upper level of any two adjacent levels in the Trie tree is no greater than the number of attribute information corresponding to the lower level of the two levels.

[0085] Specifically, the number of mutually incompatible attribute information represented by the nodes at each layer in the Trie tree can be counted as the number of attribute information corresponding to each layer. Subsequently, the number of attribute information corresponding to two adjacent layers in the Trie tree can be compared in order from upper to lower layers. If the number of attribute information corresponding to the upper layer of the two layers is greater than the number of attribute information corresponding to the lower layer of the two layers, the layers can be swapped in order.

[0086] As Figure 4Taking the Trie tree shown as an example, in this Trie tree, except for the root node, the nodes located on the first layer are nodes representing attribute information of attribute type A. Since the number of mutually unmatched attribute information in the attribute information of attribute type A is 1, it can be determined that the number of attribute information corresponding to the first layer is 1; the nodes located on the second layer are nodes representing attribute type B. Since the number of mutually unmatched attribute information in the attribute information of attribute type B is 3, it can be determined that the number of attribute information corresponding to the second layer is 3; the nodes located on the third layer are nodes representing attribute type C. Since the number of mutually unmatched attribute information in the attribute information of attribute type C is 2, it can be determined that the number of attribute information corresponding to the third layer is 2.

[0087] Subsequently, in order from the upper layer to the lower layer, first, the number of attribute information corresponding to the upper and lower adjacent first layer (upper layer) and second layer (lower layer) can be compared. Since the number of attribute information corresponding to the first layer is less than the number of attribute information corresponding to the second layer, the upper and lower order of the first and second layers can be omitted. Then, the number of attribute information corresponding to the upper and lower adjacent second layer (upper layer) and third layer (lower layer) can be compared. Since the number of attribute information corresponding to the second layer is greater than the number of attribute information corresponding to the third layer, the upper and lower order of the second and third layers can be exchanged, that is, the node originally located on the third layer is moved to the second layer, and the node originally located on the second layer is moved to the third layer, and the connection relationship between the nodes is adjusted at the same time. After completing the upper and lower order exchange of the second and third layers, the new Trie tree is as follows: Figure 5 shown.

[0088] In some embodiments, the structure of the Trie tree serving as the attribute sub-index can be optimized periodically at regular intervals. Alternatively, since the attribute sub-index is typically stored in memory, the memory space occupied by the index (including the attribute sub-index, or including the attribute sub-index and the vector sub-index loaded into memory) can be checked in real time or periodically to see if it exceeds a preset threshold. If the memory space occupied by the index exceeds the threshold, the structure of the Trie tree can be optimized, i.e., the number of mutually unmatched attribute information in the attribute information represented by the nodes at each layer in the Trie tree can be counted as the number of attribute information corresponding to each layer.

[0089] In the technical solution provided by one or more embodiments of the present application, each data set in the database may contain vector data and attribute information subsets, and the index may include an attribute sub-index established based on the attribute information subsets contained in the data sets in the database, for indicating the vector sub-index associated with each attribute information subset, and a vector sub-index established based on the vector data contained in the data sets in each data set group, for indicating the data sets containing each vector data, wherein the attribute information subsets contained in the data sets in each data set group match; accordingly, when a query request for the database is obtained, the attribute sub-index can be searched first based on the target attribute information subset in the query request to determine the target vector sub-index associated with the target attribute information subset, and then the target vector data in the query request can be searched in the target vector sub-index to determine the target data set containing the target vector data, and the query result corresponding to the query request can be determined based on the target data set.

[0090] Using the above method, for a data set containing vector data and attribute information subsets stored in a database, data retrieval can be performed by combining vector indexes established based on vector data and attribute indexes established based on attribute information, thereby realizing vector mixed retrieval.

[0091] Corresponding to the aforementioned method embodiments, the present application also provides device embodiments.

[0092] Please refer to Figure 7 , Figure 7 7 is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708 and a non-volatile memory 710, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of the present application do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0093] Please refer to Figure 8 , Figure 8 It is a block diagram of a vector hybrid retrieval device shown in an exemplary embodiment of the present application.

[0094] The above-mentioned vector hybrid retrieval device can be applied to Figure 7The device shown is used to implement the technical solution of the present application. It should be noted that each data set in the database contains vector data and attribute information subsets; the index in the database includes an attribute sub-index established based on the attribute information subset, and a vector sub-index established based on the vector data contained in the data sets in each data set group; the attribute information subsets contained in the data sets in each data set group match; the attribute sub-index is used to indicate the vector sub-index associated with each attribute information subset; the vector sub-index is used to indicate the data set containing each vector data. The device may include: The query acquisition module 802 acquires a query request for the database; wherein the query request includes target vector data and a target attribute information subset; An attribute search module 804 searches the attribute sub-index based on the target attribute information subset in response to the query request to determine a target vector sub-index associated with the target attribute information subset; The vector search module 806 searches the target vector sub-index based on the target vector data to determine a target data set containing the target vector data, and determines a query result corresponding to the query request based on the target data set.

[0095] In some embodiments, the attribute sub-index is an index based on a Trie tree; each node in the Trie tree represents each attribute information; the database further stores a correspondence between leaf nodes in the Trie tree and the vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree is a vector sub-index associated with a subset of attribute information consisting of nodes on a path from a root node in the Trie tree to each leaf node; The searching in the attribute sub-index based on the target attribute information subset to determine the target vector sub-index associated with the target attribute information subset includes: Based on the target attribute information subset, searching in the Trie tree to determine a path from the root node to the target leaf node; wherein the attribute information subset composed of the nodes on the path is the target attribute information subset; The vector sub-index corresponding to the target leaf node is determined as the target vector sub-index associated with the target attribute information subset.

[0096] In some embodiments, nodes at the same level in the Trie tree represent attribute information of the same attribute type; The device further comprises: A statistics module, which counts the number of mutually unmatched attribute information in the attribute information represented by the nodes at each layer in the Trie tree as the number of attribute information corresponding to each layer; A comparison module compares the number of attribute information corresponding to two adjacent layers in the Trie tree in order from upper layer to lower layer; The optimization module exchanges the upper and lower layers in order of the two layers if the number of attribute information corresponding to the upper layer of the two layers is greater than the number of attribute information corresponding to the lower layer of the two layers.

[0097] In some embodiments, the attribute sub-index is stored in memory; The device further comprises: A detection module detects whether the memory space occupied by the index exceeds a preset threshold; The statistics module specifically performs: If the memory space occupied by the index exceeds the threshold, the number of mutually unmatched attribute information in the attribute information represented by the nodes of each layer in the Trie tree is counted as the number of attribute information corresponding to each layer.

[0098] In some embodiments, the vector index is stored on disk; The searching in the target vector sub-index based on the target vector data includes: The target vector sub-index in the disk is loaded into the memory, and based on the target vector data, the target vector sub-index in the memory is searched.

[0099] In some embodiments, the apparatus further comprises: A batch query acquisition module is configured to acquire a batch query request for the database; wherein the batch query request includes multiple query requests belonging to the same batch; a batch attribute search module, in response to the batch query request, searching in the attribute sub-index based on the attribute information subset in each query request to determine a vector sub-index associated with the attribute information subset in each query request; a query grouping module, grouping query requests in the batch query requests according to associated vector sub-indexes, so as to classify query requests with the same associated vector sub-index into the same group; a batch vector search module that loads vector sub-indexes associated with each query request group from disk into memory, and searches the vector sub-indexes associated with the query request group in memory based on the vector data in the query requests in the query request group, to determine a data set containing the vector data in each query request in the query request group as a query result corresponding to each query request in the query request group; The result processing module aggregates the query results corresponding to the respective query requests in the batch query request to determine the batch query result corresponding to the batch query request.

[0100] For the device embodiments, they basically correspond to the method embodiments, so for relevant details, please refer to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of this application.

[0101] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0102] In a typical configuration, a computer includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0103] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0104] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0105] It should be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, commodity, or apparatus comprising the element.

[0106] The above description is of specific embodiments of the present application. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0107] The terms used in one or more embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present application. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. The term "and / or" refers to and includes any or all possible combinations of one or more of the associated listed items.

[0108] The terms "one embodiment," "some embodiments," "example," "specific example," or "one implementation" used in one or more embodiments of the present application mean that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic descriptions of these terms do not necessarily refer to the same embodiment. Moreover, the specific features or characteristics described can be combined in a suitable manner in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics in different embodiments can be combined without conflict.

[0109] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of", "when the time of" or "in response to determining".

[0110] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

[0111] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A database system, comprising a database; wherein: Each data set in the database includes vector data and attribute information subsets; The index in the database includes an attribute sub-index established based on the attribute information subset and a vector sub-index established based on the vector data contained in the data sets in each data set group; the attribute information subset contained in the data sets in each data set group matches; The attribute sub-index is used to indicate a vector sub-index associated with each attribute information subset; the vector sub-index is used to indicate a data set containing each vector data.

2. According to the database system of claim 1, the attribute sub-index is an index based on a Trie tree; each node in the Trie tree represents each attribute information; the database also stores the correspondence between the leaf nodes in the Trie tree and the vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree is a vector sub-index associated with a subset of attribute information consisting of nodes on the path from the root node in the Trie tree to each leaf node.

3. According to the database system of claim 2, the nodes located at the same layer in the Trie tree represent attribute information of the same attribute type; the number of attribute information corresponding to the upper layer of the two adjacent layers in the Trie tree is not greater than the number of attribute information corresponding to the lower layer of the two layers; the number of attribute information corresponding to each layer in the Trie tree is the number of attribute information that does not match each other in the attribute information represented by the nodes of each layer.

4. The database system according to claim 3, wherein the attribute sub-index is stored in memory; and / or the vector sub-index is stored in a disk.

5. A vector hybrid retrieval method; each data set in a database comprises vector data and an attribute information subset; an index in the database comprises an attribute sub-index established based on the attribute information subset and a vector sub-index established based on the vector data contained in the data sets in each data set group; the attribute information subsets contained in the data sets in each data set group match; The attribute sub-index is used to indicate the vector sub-index associated with each attribute information subset; The vector sub-index is used to indicate a data set containing each vector data; The method comprises: Obtaining a query request for the database; wherein the query request includes target vector data and a subset of target attribute information; In response to the query request, searching in the attribute sub-index based on the target attribute information subset to determine a target vector sub-index associated with the target attribute information subset; Based on the target vector data, a search is performed in the target vector sub-index to determine a target data set containing the target vector data, and based on the target data set, a query result corresponding to the query request is determined.

6. The method according to claim 5, wherein the attribute sub-index is an index based on a Trie tree; each node in the Trie tree represents a piece of attribute information; the database further stores a correspondence between leaf nodes in the Trie tree and the vector sub-index; the vector sub-index corresponding to each leaf node in the Trie tree is a vector sub-index associated with a subset of attribute information consisting of nodes on a path from a root node in the Trie tree to each leaf node; The searching in the attribute sub-index based on the target attribute information subset to determine the target vector sub-index associated with the target attribute information subset includes: Based on the target attribute information subset, searching in the Trie tree to determine a path from the root node to the target leaf node; wherein the attribute information subset composed of the nodes on the path is the target attribute information subset; The vector sub-index corresponding to the target leaf node is determined as the target vector sub-index associated with the target attribute information subset.

7. The method according to claim 6, wherein the nodes at the same layer in the Trie tree represent attribute information of the same attribute type; The method further comprises: Counting the number of mutually unmatched attribute information among the attribute information represented by the nodes at each layer in the Trie tree as the number of attribute information corresponding to each layer; Comparing the numbers of attribute information corresponding to two adjacent layers in the Trie tree in order from upper layer to lower layer; If the number of attribute information corresponding to the upper layer of the two layers is greater than the number of attribute information corresponding to the lower layer of the two layers, the upper and lower layers are sequentially swapped for the two layers.

8. The method according to claim 7, wherein the attribute sub-index is stored in a memory; Before counting the number of mutually unmatched attribute information among the attribute information represented by the nodes at each layer in the Trie tree as the number of attribute information corresponding to each layer, the method further includes: Detecting whether the memory space occupied by the index exceeds a preset threshold; If the memory space occupied by the index exceeds the threshold, the number of mutually unmatched attribute information in the attribute information represented by the nodes of each layer in the Trie tree is counted as the number of attribute information corresponding to each layer.

9. The method according to claim 5, wherein the vector index is stored on a disk; The searching in the target vector sub-index based on the target vector data includes: The target vector sub-index in the disk is loaded into the memory, and based on the target vector data, the target vector sub-index in the memory is searched.

10. The method according to claim 9, further comprising: Obtaining a batch query request for the database; wherein the batch query request includes multiple query requests belonging to the same batch; In response to the batch query request, searching in the attribute sub-index based on the attribute information subset in each query request to determine a vector sub-index associated with the attribute information subset in each query request; Grouping query requests in the batch query requests according to associated vector sub-indexes, so as to classify query requests with the same associated vector sub-index into the same group; Loading the vector sub-index associated with each query request group from the disk into the memory, and searching the vector sub-index associated with the query request group in the memory based on the vector data in the query requests in the query request group, to determine a data set containing the vector data in each query request in the query request group as the query result corresponding to each query request in the query request group; The query results corresponding to the respective query requests in the batch query request are aggregated to determine a batch query result corresponding to the batch query request.

11. A vector hybrid retrieval device; each data set in a database comprises vector data and an attribute information subset; an index in the database comprises an attribute sub-index established based on the attribute information subset and a vector sub-index established based on the vector data contained in the data sets in each data set group; the attribute information subsets contained in the data sets in each data set group match; The attribute sub-index is used to indicate the vector sub-index associated with each attribute information subset; The vector sub-index is used to indicate a data set containing each vector data; The device comprises: A query acquisition module, which acquires a query request for the database; wherein the query request includes target vector data and a target attribute information subset; an attribute search module, in response to the query request, searching in the attribute sub-index based on the target attribute information subset to determine a target vector sub-index associated with the target attribute information subset; The vector search module searches the target vector sub-index based on the target vector data to determine a target data set containing the target vector data, and determines a query result corresponding to the query request based on the target data set.

12. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 5 to 10 by running the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the method according to any one of claims 5 to 10 is implemented.

Images